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FOREWORD 


A  concern  of  the  U.S.  Army  intelligence  Center  and  School 
is  the  development  of  methods  for  assessing  military  intelli¬ 
gence.  The  Implementation  Guide  for  Assessing  Military  Intel¬ 
ligence  provides  the  division  commander  and  G2  a  low  cost  method 
of  determining  how  well  intelligence  is  meeting  the  needs  of  the 
command.  While  the  guide  is  primarily  directed  at  assessment 
within  an  operational  environment,  it  also  has  implications  for  ■. ' 

instruction  within  the  U.S.  Army  Intelligence  Center  and  School. jv 

« 

The  Implementation  Guide  is  a  result  of  research  conducted 
within  the  Fort  Huachuca  Field  Unit  of  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  under  a  contract 
to  Science  Applications  International  Corporation.  The  project 
was  sponsored  by  the  Directorate  of  Training  and  Doctrine,  U.S. 
Army  Intelligence  Center  and  School,  in  a  Letter  of  Agreement 
dated  14  March  1986.  The  Implementation  Guide  was  provided  to 
the  Commander,  U.S.  Army  Intelligence  Center  and  School. 


EDGAR  M.  JOHNSON 
Technical  Director 


v 


ACKNOWLEDGMENT 


The  authors  are  grateful  to  the  4th  Infantry  Division,  Fort 
Carson,  Colorado  for  permitting  us  to  use  its  time  and  personnel 
to  evaluate  this  methodology  during  a  command  post  exercise.  The 
lessons  learned  from  implementing  the  methodology  in  the  4th 
Infantry  Division  exercise  have  been  incorporated  throughout  this 
guide. 


XX 

FIELD  TRAINING  EXERCISE 


•Observer-Participant  (See  Section  2,  Personnel) 


The  Assessment  Methodology  Includes  Exchanges  of 
Information  Needs,  Rating,  Diagnosis,  and  Feedback 


vii 


IMPLEMENTATION  GUIDE  FOR  ASSESSING  INTELLIGENCE  PRODUCTION 
EFFECTIVENESS 


PROLOGUE _ _ 

The  schematic  on  the  preceding  page  is  a  summary  of  the  Military 
Intelligence  Unit  Effectiveness  Assessment  Methodology.  It  in¬ 
volves  the  exchange  of  information  needs,  the  rating  of  intel¬ 
ligence  output,  the  diagnosis  of  intelligence  production  defi¬ 
ciencies,  and  feedback  to  the  commander  and  G2. 

TO  THE  COMMANDER 

As  the  commander,  you  have  many  tools  to  use  to  assess 
effectiveness  within  your  command.  The  Skill  Qualification 
Tests,  the  Army  Training  and  Evaluation  Programs,  and  your  own 
internal  assessment  programs  provide  information  you  need  to 
determine  your  command's  capabilities  to  carry  out  its  mission. 

The  method  for  assessing  intelligence  production  effectiveness 
described  in  this  guide  is  a  tool  designed  for  you  to  use  in  an 
operational  setting  to  determine  how  well  intelligence  is  meet¬ 
ing  the  needs  of  your  command.  The  method  can  be  used  in  com¬ 
mand  post  or  field  training  exercises  where  intelligence  must 
serve  many  organizational  elements  or  where  intelligence  pro¬ 
duction  is  limited.  . 

This  methodology  has  low  overhead.  It  doesn't  take  a  dedicated 
division  team  or  outside  team  of  evaluators  to  implement.  It 
will  provide  you  a  pragmatic  view  of  how  the  intelligence  staff 
is  supporting  the  divisional  command  and  control.  It  also  pro¬ 
vides  a  way  for  the  divisional  personnel  to  improve  their  abil¬ 
ity  to  communicate  their  intelligence  information  needs  and  the 
standards  used  to  judge  the  intelligence.  Finally,  it  provides 
a  feeder  mechanism  for  intelligence  training  that  will  enhance 
division  command  and  control  performance. 

TO  THE  G2 

This  Implementation  Guide  for  Assessing  Intelligence  Production 
provides  you  a  tool  for  measuring  and  assessing  how  well  your 
section  meets  the  intelligence  needs  of  the  command.  It  per¬ 
mits  you  to  look  at  performance  inside  the  intelligence  produc¬ 
tion  system-based  intelligence  requirements,  standards,  and 
assessments  from  the  division  user.  It  provides  you  a  method 
for  isolating  production  areas  that  need  improving  from  those 
that  don't  and  can  give  you  insight  into  how  to  improve  the 
usefulness  of  intelligence  for  the  user.  In  addition,  the 
method  will  identify  functional  problems  outside  the  intelli¬ 
gence  infrastructure  that  may  have  been  incorrectly  attributed 
to  Military  Intelligence. 
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IMPLEMENTATION  GUIDE  FOR  ASSESSING  INTELLIGENCE 
PRODUCTION  EFFECTIVENESS 

SECTION  1 


INTRODUCTION 

The  methodology  presented  in  the  Implementation  Guide  is  driven 
by  the  useri  of  intelligence.  It  depends  on  users  identifying 
their  intelligence  requirements  and  judging  how  well  the 
requirements  have  been  met.  Figure  1  shows  the  steps  required  in 
the  assessment.  Using  the  tools  provided,  the  user  of 
intelligence  first  identifies  and  prioritizes  his  information 
requirements.  Then,  using  a  simple  procedure,  he  determines  the 
standards  he  uses  to  judge  the  information  he  receives.  Finally, 
he  rates  the  information  he  received.  The  analysis  of  the 
ratings  results  in  a  pattern  of  intelligence  production 
performance  that  is  used  to  determine  where  deficiencies  might 
be  occurring  within  the  intelligence  production  process.  The 
ratings  are  also  used  to  develop  a  diagnostic  plan  to  determine 
the  causes  for  the  deficiencies.  Then  the  intelligence 
production  system  is  diagnosed.  The  result  of  the  diagnosis, 
along  with  the  patterns  of  performance  and  the  ratings,  form  the 
basis  of  the  feedback  to  the  commander  and  G2. 

Organization  of  the  Implementation  Guide 

Section  2  covers  who  is  involved  in  the  assessment.  It 
describes  the  qualifications  the  participants  should  have  and  the 
roles  they  play  in  the  assessment.  Section  3  discusses  the 
administrative  procedures  necessary  to  plan  and  implement  the 
assessment.  The  next  four  sections,  identified  in  Figure  1,  cover 
the  steps  necessary  to  carry  out  the  asses?  .ent,  diagnosis,  and 
feedback.  Each  step  includes  the  procedures  and  forms  necessary 
to  carry  out  the  step  and  administrative  concerns  relevant  to  the 
completion  of  the  step.  Section  4  covers  the  procedures 
necessary  to  identify  and  prioritize  the  information  requirements 
and  identify  standards  used  to  assess  intelligence.  Section  5 
discusses  the  steps  the  user  must  take  to  rate  the  information 
received  from  intelligence.  Section  6  provides  suggestions  for 
analyzing  the  user  ratings.  Section  7  provides  guidelines  for 
planning  and  diagnosing  intelligence  production  deficiencies. 
Section  8  offers  suggestions  for  providing  feedback. 
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STEP  1 
Section  4 


STEP  2 
Section  4 


STEP  3 
Section  5 


STEP  4 
Section  6 


STEP  5 
Section  6 


STEP  6 
Section  7 


STEP  7 
Section  7 


STEP  8 
Section  8 


Identify  and  Prioritize 
Information  Requirements 

\ 

Determine  Standards  for 
the  Information 

i 

Rating  the  Information 
Received 

I 

Analyze  the  Ratings 

1 

Determine  Where  Deficiencies 
Might  Have  Occurred 

i 

Develop  a  Diagnostic  Plan 

\ 

Diagnose  the  Intelligence 
Information  Processing  System 

j 

Provide  Feedback 


Figure  1.  Overview  of  the  steps  in  the  intelligence  assessment  methodology. 
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SECTION  2 


PERSONNEL 

The  assessment  methodology  is  implemented  by  a  team  of  soldiers 
each  having  specific  responsibilities,  some  which  are  done 
individually  and  others  done  as  part  of  the  team  effort.  This 
section  identifies  the  team  members,  their  roles,  and  general 
qualifications.  The  team  has  three  roles:  the  administration  of 
the  assessment,  the  assessment  of  Military  Intelligence 
performance,  and  the  diagnosis  of  deficiencies  in  the  production 
of  intelligence.  The  size  of  the  team  is  as  large  or  as  small  as 
is  warranted  by  the  exercise  and  assessment  objectives. 

A.  COMMANDER  AND  G2 . 

Although  the  commander  and  the  G2  may  not  be  involvt  in  the 
actual  assessment,  they  are  critical  to  the  assessment.  They  have 
made  the  decision  to  do  the  assessment  and  it  is  carried  out 
under  their  authority.  The  results  of  the  assessment  are 
provided  to  the  G2  and  the  commander  through  the  G2.  Therefore, 
each  has  a  vested  interest  to  ensure  the  assessment  is  conducted 
as  completely,  effectively,  and  efficiently  as  possible. 

B.  ASSESSMENT  TEAM  CHIEF. 

Role:  The  assessment  team  chief  manages  the 

assessment.  He  is  the  administrative 
component  of  the  assessment  team.  He 
is  responsible  for  assembling  and 
training  the  other  team  members.  With 
assistance  of  the  other  team  members, 
he  determines  how  the  assessment 
methodology  can  best  be  implemented, 
ensures  the  other  teams  members  have 
the  time  and  resources  to  carry  out 
their  functions,  and  serves  as  the 
liaison  for  the  team. 

Qualifications:  The  assessment  team  chief  should  be 

knowledgeable  of  the  division 
intelligence  operations  and  personnel. 
He  should  be  able  to  easily 
communicate  with  personnel  at  all 
levels  of  the  division  command  and 
within  its  operations. 

Suggested  Candidate:  Deputy  G2  or  Military  Intelligence 

Battalion  Executive  Officer 
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C.  INTELLIGENCE  USERS. 


Role: 


Qualifications: 


Suggested  Candidate: 


D.  DIAGNOSTICIANS. 
Role: 

Qualifications: 


Suggested  Candidate: 


The  users  of  intelligence  are  the 
soldiers  who  will  assess  how  well  the 
intelligence  information  production 
system  is  working.  They  are  active 
participants  in  the  exercise  and  at 
the  same  time  assess  how  well  the 
information  received  met  their 
intelligence  needs.  Because  they  are 
active  participants,  they  will  be 
referred  to  as  OBSERVER-PARTICIPANTS 
in  the  rest  of  this  guide. 

Since  the  observer-participants  are 
active  participants  in  the  exercise, 
it  is  required  they  have  some 
knowledge  and  experience  in  the 
mission  related  role  they  will  be 
playing. 

Officers  from  G3  Operations  and  Plans, 
the  Fire  Support  Element,  Targeting, 
the  Division  Airspace  Management 
Element  (DAME) ,  etc.  Representatives 
of  subordinate  command  echelons  may  be 
included. 


The  diagnostician  will  try  to 
determine  the  reasons  for  the 
deficiencies  identified  by  the 
observer-participants . 

The  diagnostician  must  have  experience 
in  Military  Intelligence  and  know 
intelligence  doctrine.  He  must  know 
or  learn  the  Standard  Operating 
Procedures  for  the  division  G2  section 
being  assessed. 

Any  senior  350B  Military  Intelligence 
Warrant  Officer. 
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SECTION  3 


ADMINISTRATIVE  PROCEDURES 

While  the  administrative  burden  for  the  assessment  falls  on  the 
assessment  team  chief,  all  members  of  the  team  should  be  aware  of 
the  overall  requirements  of  the  methodology.  This  will  help  each 
person  better  understand  the  various  roles  and  how  everyone 
contributes  to  the  assessment. 

The  administrative  procedures  concern  those  involved  in  planning 
and  organizing  the  assessment  and  those  that  must  be  carried  out 


during  the  exercise. 

A.  SELECTING  THE  TEAM  MEMBERS. 
Assessment  Team  Chief: 

Observer-Participants : 


EXAMPLE 


The  commander  or  his  designate 
must  make  this  selection. 

The  assessment  team  chief  should 
prepare  a  list  of  recommended 
observer-participants 
representative  of  key  division 
staff  agencies  and  functions. 
Approval  of  the  list  should  be 
made  by  the  commander  or  his 
designee.  How  many  and  who 
should  be  on  the  list  depends  on 
the  purpose  and  desired  scope  of 
the  assessment. 

If  the  assessment  is  to  be 
conducted  in  a  short  duration — 2 
5  days — division  headquarters  and 
staff  command  post  exercise 
intended  to  shake  out  staff 
planning  procedures,  the 
observer-participants  may  be 
limited  to  one  from  G3  Plans  and 
one  from  G3  OPS. 

If  the  assessment  is  to  be 
conducted  during  a  Divisional 
Field  Training  Exercise  with  a 
control  cell  and  all  organization 
elements,  maneuver  and  staff, 
participating  or  conducting 
operations,  then  numerous 
observer-participants  are  needed 
to  ensure  assessment  of  the 
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broadest  aspects  of  intelligence 
production  and  support.  All 
staff  agencies,  selected  special 
staff,  and  alternate  command 
posts  should  be  considered. 

Diagnostician:  The  team  chief  should  submit  a 

name  for  approval  to  the 
commander  or  his  designee  based 
on  the  qualifications  mentioned 
previously.  One  diagnostician  is 
enough  unless  the  exercise 
environment  is  so  extensive  that 
it  might  require  full  shift 
coverage  for  diagnosis. 

Upon  approval,  team  members  should  be  made  available  for  the 
assessment  training  at  least  two  weeks  before  the  start  of  the 
exercise.  Two  weeks  is  a  rule  of  thumb,  but  team  members  must 
be  trained  and  exercise  administrative  details  worked  out  before 
the  assessment  can  be  implemented. 

In  addition,  the  assessment  team  chief  must  perform  the  following 
administrative  duties: 

B.  BRIEFING  THE  KEY  STAFF. 

While  the  commander  may  have  informed  his  staff  an  assessment 
will  be  done,  it  is  a  good  idea  to  provide  key  staff  members 
more  detail.  Since  headquarters  or  unit  soldiers  are  being  used 
as  observer-participants,  it  is  only  right  that  the  key  staff 
members  know  who  from  their  staff  are  participating,  what  is 
going  to  be  done,  and  how  much  time  it  is  going  to  take.  In  fact, 
the  key  staff  should  be  consulted  for  recommendations  for  the 
observer-participants  roles. 

If  a  group  briefing  can  not  be  done,  each  key  staff  member  must 
be  told  what  the  assessment  entails.  This  may  be  done  if  and 
when  the  key  staff  is  consulted  to  provide  names  for  the 
assessment  team. 

C.  TRAINING  THE  TEAM. 

It  is  the  responsibility  of  the  team  chief  to  train  the  rest  of 
the  team.  If  this  is  the  first  time  he  has  administered  an 
assessment,  his  knowledge  will  be  as  limited  as  the  other  team 
members.  The  training  should  be  considered  a  team  effort,  with 
team  members  helping  each  other.  An  example  training-time  plan 
is  shown  in  Figure  2.  The  events  can  occur  within  the  windows 
indicated  by  the  triangles,  but  no  later.  To  facilitate  such  a 
situation,  all  team  members  need  to  receive  copies  of  the 
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EVENT 


Figure  2.  An  example  schedule  for  observer-participant  training. 


implementation  guide  well  before  the  scheduled  training  period. 

By  training  time,  everyone  should  be  prepared  to  discuss  the 
assessment  procedures. 

NO  LESS  THAN  8  HOURS  SHOULD  BE  ALLOTTED  FOR  ASSESSMENT  TEAM 
TRAINING  TIME.  THE  SUCCESS  OF  THE  ASSESSMENT  IS  BUILT  UPON  THE 
ACCOMPLISHMENT  OF  THE  PREASSESSMENT  TRAINING. 

The  goal  of  the  training  is  to  ensure  that  everyone — team  chief, 
observer-participants,  and  diagnostician — knows  what  forms  to  use 
and  when  and  how  to  use  them.  If  possible,  a  practical  exercise 
should  be  part  of  the  training  to  give  the  observer- 
participants  experience  filling  in  the  forms.  Also,  the 
observer-participants  need  to  know  how  the  data  they  produce  will 
be  analyzed  so  that  it  can  be  used  by  the  diagnostician. 

Finally,  everyone  needs  to  know  the  administrative  procedures 
that  will  be  followed  during  the  exercise.  These  procedures  need 
to  be  established  during  the  team  training  meeting. 

D.  CONDUCTING  A  PRE-EXERCISE  CONFERENCE. 

The  entire  team  must  meet  before  the  exercise  begins.  The 
purposes  of  this  meeting  are  to  review  the  assessment 
administrative  procedures  that  will  be  in  effect,  hand  out  and 
ensure  everyone  has  enough  forms,  and  obtain  the  initial 
information  requirements  profile  and  benchmarks  from  the 
observer-participants . 

E.  PREPARING  ASSESSMENT  MATERIALS. 

All  team  members  must  have  a  copy  of  the  Implementation  Guide 
for  their  training.  All  the  forms,  in  the  appropriate 
quantities,  needed  by  the  observer-participants  and 
diagnostician  should  be  provided  at  the  preconference  exercise. 
Blank  forms  are  in  the  appendices  and  can  be  reproduced  in 
whatever  quantity  is  necessary. 

There  are  administrative  assessment  procedures  for  which  the 
team  chief  should  get  input  from  the  observer-participants.  The 
details  of  these  procedures  need  to  be  determined  during  the 
training  session. 

F.  RATING  MI  PERFORMANCE. 

The  team  should  decide  the  best  time  in  each  exercise  day  to  rate 
MI  performance.  The  best  time  may  depend  on  the  nature  and 
length  of  the  exercise,  as  well  as  the  objectives  for  the 
assessment.  Whatever  the  time  frame  for  rating,  all  observer- 
participants  must  rate  MI  performance  within  the  same  time 
window. 
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In  determining  when  to  rate,  the  team  should  consider: 

The  amount  of  intelligence  each  observer- 
participant  will  be  getting  within  the  selected 
time  interval.  The  interval  should  not  be  so  long 
that  it  would  be  difficult  to  remember  the  intelligence 
that  had  been  used,  nor  so  short  that  there  is  little  to 
assess. 

The  natural  breaks  which  provide  good  assessment  points. 
For  example,  at  the  end  of  shift  or  before  the  beginning 
of  the  next  shift  may  be  good  times  to  take  the  time 
necessary  to  do  the  rating.  Also,  natural  phases  of  the 
exercise  are  meaningful  points  to  assess.  For  example, 
rating  after  the  planning  phase,  after  an  attack,  etc., 
assess  how  well  intelligence  was  provided  for  that  phase. 

The  observer-participants  duties  and  responsibilities 
during  the  exercise.  It  takes  time  to  rate.  Even  though 
the  observer-participant  is  committed  to  assessing,  he  is 
still  an  active  player  and  the  assessment  can  not 
interfere  with  the  exercise. 

If  it  becomes  necessary  to  change  the  rating  period,  the 
assessment  team  chief  must  inform  all  members  of  the  team  so 
they  can  rate  within  the  new  time  window. 

G.  DATA  COLLECTION. 

Ratings  must  be  analyzed  by  the  team  chief  and  diagnostician  in 
order  for  the  diagnostician  to  carry  out  his  function.  The  team 
should  determine  the  time  window  to  complete  the  rating  and  the 
best  means  to  get  the  ratings  to  the  team  chief.  Most  likely, 
how  the  ratings  are  delivered  will  depend  on  the  physical  layout 
of  the  exercise  and  the  distribution  of  observer-participants 
within  it. 

H.  QUALITY  CONTROL  OF  THE  DATA. 

The  success  of  the  assessment  depends  on  the  quality  of  the  data 
that  is  collected.  Each  individual  on  the  team  knows  what  is 
required  of  him  and  how  to  accomplish  his  responsibilities.  Each 
is  responsible  for  the  quality  control  of  their  own  data.  If  the 
diagnostician  or  the  observer-participants  run  into  any  problems 
in  data  collection  that  they  cannot  resolve,  they  need  to  take 
the  problem  to  the  team  chief  immediately. 

Additional  administrative  concerns  are  addressed  later  in  each 
section . 
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SECTION  4 


INFORMATION  REQUIREMENTS 

The  assessment  of  MI  information  production  effectiveness  is 
based  on  how  well  MI  has  satisfied  the  information  requirements 
of  the  observer-participant.  The  first  step  in  the  assessment  is 
to  determine  the  information  requirements  of  the  various 
observer-participants.  This  is  done  by  the  observer-participants 
assigning  weights  to  the  information  items  in  a  list  of 
information  requirements.  The  weights  produce  an  information 
requirements  profile.  Once  weighed,  the  observer-participant 
determines  the  standards  he  will  use  to  judge  the  information  he 
receives  (Step  2) . 

A.  INFORMATION  REQUIREMENTS  PROFILE. 

A  copy  of  a  blank  Information  Requirements  Profile  is  shown  in 
Figure  3.  A  copy  of  the  profile,  the  definitions  of  the  terms  in 
the  profile,  and  a  complete  set  of  directions  for  weighing  the 
items  are  in  Appendix  A. 

B.  DESCRIPTION  OF  THE  PROFILE. 

In  the  box  at  the  top  left  corner  of  the  Information  Requirements 
Profile  are  abbreviated  instructions  on  how  to  complete  the 
profile.  The  first  time  the  form  is  completed,  the  full  set  of 
directions  (Appendix  A-2)  need  to  be  used.  After  that,  the 
abbreviated  set  can  be  used. 

Below  the  instructions  in  the  left  hand  vertical  column  is  the 
list  of  information  requirements.  The  column  is  divided  into 
four  major  sections: 

BATTLEFIELD  AREA 
ENEMY  SITUATION 
ENEMY  COURSES  OF  ACTION 
OPERATIONS  SECURITY 

These  headings  characterize  the  kind  of  information  item  found 
in  the  section.  They  also  provide  the  most  general  level  of 
definition. 

Each  of  the  major  sections  are  broken  down  into  sub-sections. 

For  example,  BATTLEFIELD  AREA  has  three  kinds  of  information 
items,  weather,  terrain,  and  battlefield  area  conditions.  The 
subsections  also  have  general  definitions. 

Each  subsection  contains  the  information  items.  Each  item  has  a 
more  specific  definition  than  the  major  sections  or  subsections. 
The  individual  information  items  will  be  weighed. 
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INFORMATION  REQUIREMENTS  PROFILE 


In  the  center  column,  the  observer-participant  places  the 
weights  he  assigns  the  information  item. 

The  "special  notes  column"  is  not  part  of  the  weighing  procedure. 
The  observer-participant  can  identify  specific  requirements 
concerning  that  item. 

EXAMPLE  An  observer-participant  may  not  only  be 

interested  in  the  weather  situation  in  general, 
but  may  also  be  interested  if  there  will  be  30 
mph  winds  between  0400  and  0600.  He  would  make 
that  note  in  the  special  notes  column. 

C.  STEP  1.  PROCEDURES  FOR  IDENTIFYING  AND  PRIORITIZING 
INFORMATION  REQUIREMENTS . 

1.  Reviewing  the  Information  Items. 

The  observer-participant  reviews  each  item  and  its 
definition  (Appendix  A-3)  to  ensure  he  knows  their 
meaning.  The  definitions  should  be  referred  to  during 
weighing  when  the  observer-participant  has  any  doubt 
about  the  meaning  of  an  item.  The  review  is  an 
important  step  because  it  helps  establish  consistent 
meanings  for  an  observer-participant  and  between 
different  observer-participants. 

2.  Initial  Weighing. 

All  items  are  reviewed  as  to  their  importance  within  the 
specific  rating  period.  For  example,  if  the  rating 
period  is  the  planning  phase,  the  importance  of  the  items 
would  be  relative  to  the  planning  phase. 

The  items  that  are  most  important  are  given  100  points. 

If  there  are  any  items  the  observer-participant  does  not 
want  during  that  period,  they  are  given  a  0.  There  is  no 
limit  to  the  number  of  items  that  can  be  given  100  or  0. 

EXAMPLE  As  seen  in  Figure  4,  the  most  important  items 

and  the  items  not  needed  have  been  identified. 
Seven  items  were  identified  as  being  most 
important,  i.e.,  given  100  points.  Two  items 
were  not  needed,  i.e.,  given  0  points.  The 
items  without  assigned  weights  are  needed,  but 
are  less  important  than  those  having  the  100. 
Every  remaining  item  in  the  requirements  list 
must  have  a  weight  from  0  to  100. 
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ENEMY  COURSES  OF  ACTION 


INFORMATION  REQUIREMENTS  PROFILE 


POSITION. 


3.  Determining  Relative  Weights. 

The  remaining  items,  one  at  a  time  are  compared  to  the 
items  given  the  weight  of  100.  The  question  which  needs 
to  be  answered  is,  how  important  is  that  item,  compared 
to  the  most  important  items.  The  item  is  given  points 
accordingly.  For  example,  if  the  item  was  half  as 
important  it  would  be  given  50;  if  the  item  was  almost  as 
important,  but  not  quite,  it  might  be  given  a  90;  if  the 
item  was  needed  but  not  very  important  in  relation  to  the 
most  important  items,  it  might  be  given  a  10  or  15;  and 
if  the  item  is  as  important  as  the  other  100s,  it  is 
given  100. 

4.  Review  of  Weights. 

The  weights  need  to  be  reviewed  to  ensure  that  all  items 
have  been  weighed  and  the  reviewer  has  a  chance  to  look 
at  all  the  weighings  as  a  group.  Any  changes  to  the 
weights  can  be  made  at  this  time,  as  illustrated  in 
Figure  5. 

D.  IMPORTANCE  OF  THE  WEIGHING. 

The  weights  are  used  in  the  assessment  procedure  to  determine 
how  well  MI  has  met  the  observer-participant's  needs.  The 
weights  may  also  be  used  to  determine  priorities  for  diagnosing 
the  information  production  function. 

E.  STEP  2.  DETERMINING  THE  STANDARDS  FOR  THE  INFORMATION. 

It  is  necessary  for  the  observer-participant  to  determine  the 
standards  he  will  use  to  assess  the  information  he  receives  from 
intelligence.  This  is  done  by  identifying,  on  a  series  of 
scales,  the  point  where  acceptable  information  becomes 
unacceptable.  This  point  is  the  benchmark  for  later  determining 
MI  effectiveness.  There  are  five  dimensions  which  can  be  used  to 
assess  information:  timeliness,  frequency,  operational 
perspective,  clarity,  and  completeness. 

F.  DESCRIPTION  OF  THE  DIMENSIONS  AND  SCALES. 

The  five  dimensions  with  their  scale  and  the  directions  for 
setting  benchmarks  are  found  in  Appendix  A.  The  timeliness 
dimension  and  scale  is  shown  in  Figure  6.  Since  each  of  the 
scales  for  the  five  dimensions  are  in  the  same  format,  only  one 
is  described.  At  the  top  of  the  page  is  the  name  of  the 
dimension,  Timeliness,  in  this  case.  Next  is  the  definition  and 
explanation 
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INFORMATION  REQUIREMENTS  PROFILE 


INSTRUCTIONS: 

NAME 

1 .  Assign  zero  to  any  information  Hama  you  do  not  want. 

2.  Assign  100  to  your  most  important  information  Hama. 

3.  Assign  a  number  between  0  and  100  to  remaining  Items 
to  reflect  the  relative  importance  of  the  Hern 

POSITION 

DATE 

4.  List  specific  data  items  you  want  emphasized  under 

special  notes 

IMPORTANCE  WEIGHT 

SPECIAL  NOTES 
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TIMELINESS 


TIMELINESS  -  is  a  measure  of  whether  the 
information  item  was  received  in  time  for  the  user 
to  take  action  on  it.  Timeliness  applies  to 
situations  where  there  is  no  stated  operational 
deadline  for  the  information,  but  a  window  exists 
during  which  the  user  must  take  action  based  on 
that  specific  information. 


Timeliness  Scale: 

1.  Item  received  and  user  had  ample  time  to  take 
operational  action  within  the  time  window. 

2.  Item  received  in  time  to  take  action,  but  user 
had  to  rush. 

3.  Item  received  in  time  to  take  action,  but  user 
had  to  rush  and  use  additional  resources. 


4.  Item  received  too  late  for  user  to  take  action. 

5.  Item  not  received. 


FIGURE  6.  The  timeliness  dimension  and  scale. 


of  the  dimension.  Following  the  definition  is  the  scale.  It  has 
five  levels  numbered  1  through  5.  One  (1)  on  the  scale  is  meant 
to  represent  the  best  possible  case,  the  5  is  meant  to  represent 
the  worst  case.  The  items  in  between  represent  different  degrees 
of  performance  between  the  best  and  worst  cases. 

G.  DETERMINING  THE  BENCHMARKS. 

1.  For  any  dimension  review  the  five  levels  of  performance 
indicated  by  the  scales. 

2.  Select  the  point  on  the  scale  that  best  describes  the 
minimum  level  of  performance  which  would  be  acceptable  to 
you. 

3.  Draw  a  line  under  that  scale  number. 

4.  Review  the  items  under  that  line  to  ensure  they  describe 
performance  unacceptable  to  you.  If  not,  redetermine 
the  benchmark. 
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EXAMPLE 


As  seen  in  Figure  6,  a  line  has  been  drawn 
between  Items  3  and  4.  Therefore,  any  item 
rated  4  or  5  by  the  observer-participant 
indicates  unacceptable  performance  on 
timeliness. 

H.  ADMINISTRATIVE  CONSIDERATIONS. 

Initial  weighing  and  setting  benchmarks  are  done  during  the 
preexercise  conference  and  subsequently  during  the  exercise 
within  the  time  windows  determined  by  the  assessment  team.  The 
new  weights  and  benchmarks  completed  during  the  exercise  should 
be  turned  over  to  the  team  chief  with  the  rating  form.  During 
the  exercise,  weighing  and  setting  benchmarks  will  always  take 
place  after  the  rating  and  will  be  done  in  relation  to  the  next 
rating  period. 

EXAMPLE  The  time  schedule  shown  in  Figure  7  indicates 

that  the  first  weighing  is  done  in  the 
preexercise  conference,  which  is  held  within  two 
days  of  the  exercise  start.  The  rating  period 
for  the  exercise  is  12  hours.  Within  the  window 
the  assessment  of  MI  performance  for  the 
preceding  12  hours  occurs.  When  that  is 
completed,  the  second  weighing  is  done  for  the 
next  12  hour  period.  The  cycle  continues  for 
the  48  hour  exercise  shown.  The  arrows  pointing 
to  left  indicate  the  assessment  is  for  previous 
12  hours.  In  contrast,  arrows  to  the  right 
indicate  that  weighing  is  to  be  done  in  terms 
of  the  next  12  hours. 
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SECTION  5 


RATING  MILITARY  INTELLIGENCE  PERFORMANCE 

Sometime  after  the  observer-participants  begin  to  receive  and  use 
intelligence,  they  will  rate  how  well  the  information  met  their 
requirements.  Each  item  will  be  assessed  and  rated  on  the  five 
dimensions  for  the  rating  period  previously  determined.  The 
ratings  will  be  analyzed  by  the  team  chief  and  diagnostician. 

The  analysis  will  form  the  basis  for  the  diagnosis  of  the 
intelligence  production  system  and  performance  feedback  to  the 
commander  and  G2. 

A.  PERFORMANCE  RATING  FORM. 

A  blank  copy  of  Part  1  of  the  Performance  Rating  Form  is  shown 
in  Figure  8.  Part  2  follows  the  same  format.  A  copy  of  the 
rating  form  and  instructions  on  how  to  do  the  rating  are  in 
Appendix  B. 

B.  DESCRIPTION  OF  THE  RATING  FORM. 

At  the  left  of  the  rating  form  is  the  column  containing  the  five 
dimensions  used  to  rate  the  information  items.  Each  dimension, 
timeliness,  frequency,  operational  perspective,  clarity,  and 
completeness  are  described  by  five  point  scales.  The  description 
for  each  level  of  the  scale  is  abbreviated.  The  complete  scales 
are  located  in  Appendix  A.  There  is  also  a  box  to  check  in 
order  to  identify  the  benchmark. 

The  next  column  gives  examples  of  the  kinds  of  deficiencies  that 
could  occur  for  the  dimension  on  the  left.  The  lists  are  not 
inclusive. 

The  third  column  contains  the  information  item.  These  are  the 
same  items,  in  the  same  format,  that  are  in  the  Information 
Requirements  Profile.  To  the  right  of  the  information  items  are 
the  columns  to  be  filled  in  when  rating  intelligence 
performance.  Each  column  is  headed  by  the  dimension  on  which 
the  information  item  will  be  rated. 

C.  STEP  3.  RATING  MILITARY  INTELLIGENCE  PERFORMANCE. 

1.  Review  the  Five  Scales. 

When  the  time  window  opens  for  the  first  rating  of  the 
information  items,  the  observer-participant  should  review 
the  definitions  and  scales  for  the  five  dimensions 
(Appendix  A).  On  later  ratings,  the  review  needs  to  be 
done  only  as  necessary  since  abbreviated  scales  are  on 
the  rating  forms. 


27 


Performance  Rating  Form 
(part  1  of  2) 


.4 


2.  Rate  Timeliness. 

Each  information  item  is  rated  on  timeliness  by 
determining  which  scale  value,  1  through  5,  represents 
how  well  that  information  item  was  provided.  Starting 
with  the  first  item  on  the  Performance  Rating  Form, 
weather  situation,  the  scale  value  selected  is  placed  in 
the  column  "timeliness  rating".  Each  item  on  the  list  is 
rated  for  timeliness  until  all  the  items  have  been  rated. 
If  any  of  the  items  were  not  received,  give  a  scale  value 
of  5  and  draw  a  line  across  the  page  in  the  row  for  those 
items.  If  not  received,  an  item  cannot  be  rated  on  the 
other  dimensions. 

NOTE:  An  information  item  may  have  been  received  more  than 
once  during  the  rating  period.  The  procedure 
requires  a  rating  for  the  information  item  in  general. 
Thus,  it  is  left  up  to  the  observer-participant  to 
determine  a  final  judgment  for  a  composite  of  items. 

EXAMPLE  Figure  9  is  an  example  of  the  second  page  of  the 

Performance  Rating  Form  completed  for  the 
timeliness  rating.  Note  there  were  four  items 
not  received.  They  were  given  ratings  of  5  and 
cannot  be  assessed  on  the  other  dimensions.  In 
addition,  three  items  were  rated  as  having  been 
received  too  late  for  any  action  to  be  taken, 
e.g.  given  ratings  of  4.  Two  items  were  given 
3  ratings  because  they  were  received  in  time  to 
take  action,  but  the  observer-participant  had  to 
rush  and  use  extra  resources  to  accomplish  the 
desired  follow-up  action.  All  the  other  items 
were  received  in  time  to  take  the  appropriate 
action. 

3.  Rate  Frequency. 

After  all  the  items  have  been  rated  for  timeliness,  they 
are  rated  for  frequency.  The  rating  is  done  in  the  same 
manner.  Each  information  item  is  rated  on  frequency  by 
determining  the  scale  value,  1  through  5,  which 
represents  how  well  the  item  was  provided.  Starting  with 
the  first  item  on  the  Performance  Rating  Form,  the  scale 
value  selected  is  placed  in  the  column  "frequency 
rating."  The  same  procedure  is  carried  out  for  all  the 
remaining  items,  except  those  that  were  rated  5  for 
timeliness. 
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EXAMPLE 


Frequency  ratings  have  been  added  to  the 
previous  example  (Figure  10) .  The  ratings 
indicate  that  four  items,  rated  4,  were  not 
received  often  enough,  and  one  item,  rated  2, 
came  too  often,  but  was  not  disruptive.  The 
rest  were  received  often  enough. 

4.  Rate  Operational  Perspective. 

The  procedure  for  rating  operational  perspective  is  done 
in  the  same  manner  as  the  timeliness  and  frequency 
ratings.  However,  in  addition  to  the  scale  value  being 
placed  in  the  column  "operational  perspective — rating," 
the  observer-participant  needs  to  describe  the 
deficiency.  Sample  deficiencies  are  listed  on  the 
Performance  Rating  Form.  The  deficiencies  A  through  G, 
are  examples  of  aspects  of  the  information  which  would 
result  in  an  observer  participant  being  unsure  as  to  why 
the  information  was  received.  For  example,  if  the 
information  did  not  relate  to  either  the  area  of 
operations  or  the  area  of  interest,  it  would  lack 
operational  perspective. 

If  one  of  the  sample  deficiencies  describes  the 
deficiency  in  the  item,  the  letter  from  the  sample  list 
is  put  in  the  column  "operational  perspective- 
-def iciencies. "  If  the  deficiency  in  the  item  is  not  in 
the  sample  list,  an  abbreviated  description  of  the 
deficiency  should  be  written  in  the  column. 

EXAMPLE  The  previous  example  is  further  expanded  to 

include  operational  perspective  (Figure  11) . 

One  item  is  a  4  in  that  it  didn't  relate  to 
current  or  future  operations.  The  remaining 
items  were  rated  either  2  or  3.  Two  of  the 
items  were  deficient  in  "area  of  interest." 

(Item  B  from  the  sample  Operational  Perspective 
Deficiencies  List) .  Although  not  shown  in  this 
example,  an  item  may  have  more  than  one 
deficiency,  thus  more  than  one  letter  per  item 
can  be  put  in  the  deficiency  column. 

5.  Rate  Clarity. 

The  procedure  for  rating  clarity  is  the  same  as  that  used 
to  rate  operational  perspective.  Scale  ratings  are 
selected  and  placed  in  the  appropriate  column  for  each 
item.  Deficiencies  are  identified  by  either  a  letter 
from  the  sample  deficiency  list  or  writing  in  the 
deficiency.  An  item  may  have  more  than  one  clarity 
deficiency. 
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The  previous  example  is  expanded  to  include 
clarity  (Figure  12).  Eight  of  the  items  had 
deficiencies.  Three  had  only  1,  three  had  2  and 
the  rest  had  3  deficiencies.  Deficiency  A 
"poorly  organized"  was  the  most  frequent 
deficiency. 

6.  Rate  Completeness  and  Reestablish  Benchmarks. 

Completeness  is  rated  in  the  manner  as  clarity.  Both 
scale  values  and  the  letter  representing  the  deficiency 
are  placed  in  the  appropriate  column  for  each  item.  As 
before,  an  item  may  have  more  than  one  deficiency.  When 
you  have  completed  the  rating,  be  sure  the  benchmarks  for 
each  dimension  have  been  identified. 

EXAMPLE  The  example  is  now  complete  (Figure  13) .  All  of 

the  items  have  been  rated  for  each  of  the  five 
dimensions  and  the  deficiencies,  where 
appropriate,  have  been  identified. 

NOTE  THAT  THE  BENCHMARKS  FOR  EACH  DIMENSION  ARE 
IDENTIFIED. 

7.  Quality  Control. 

After  the  Performance  Rating  Form  has  been  completed,  it 
should  be  reviewed  to  make  sure  all  the  items  have  been 
rated  and  the  deficiencies  identified.  The  form  should 
be  turned  in  to  the  team  chief  according  to  the  procedure 
determined  in  the  training  meeting.  Although  the 
observer-participants  have  previously  established  their 
benchmarks,  they  should  reestablish  them  for  each  of  the 
dimensions  for  each  rating  period,  by  marking  the 
appropriate  boxes  in  the  left  column. 

D.  ADMINISTRATIVE  CONSIDERATIONS. 

The  ratings  take  time,  but  because  of  their  importance  to  the 
assessment,  they  must  be  accomplished.  The  ratings  must  also  be 
passed  to  the  team  chief  and  diagnostician  so  that  timely 
diagnostics  and  feedback  can  be  given.  It  is  the  responsibility 
of  the  observer-participant  to  do  what  is  necessary  to  accomplish 
the  ratings.  One  hour  should  be  enough  to  complete  the  ratings, 
although  the  time  it  takes  will  vary  depending  on  the  amount  of 
information  received,  the  number  of  deficiencies,  and  the 
experience  of  the  observer-participant  in  doing  the  ratings. 

The  procedure  outlined  above  represents  one  found  to  be  fast  and 
effective.  If  the  observer-participant  develops  a  different 
procedure  which  is  comfortable  to  him,  that  is  acceptable.  The 
primary  requirements  are  accurate  and  complete  ratings. 
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Depending  on  the  objectives  and  the  extent  of  the  exercise,  it 
may  not  be  necessary  for  the  observer-participants  to  rate  all 
the  items.  There  are  other  options. 

Rating  only  deficient  items.  This  is  the  fastest 
and  easiest  option.  The  observer-participant  rates 
only  those  items  that  were  deficient.  The  rating 
can  occur  when  the  deficiency  is  noted,  or  within 
the  agreed  upon  time  window.  While  an  acceptable 
method  for  assessment  purposes,  it  does  not  provide 
data  on  what  is  being  done  well. 

Rating  only  the  most  important  information  items. 

When  this  is  done  the  weights  for  the  most 
important  items  must  be  a  recorded  on  the  rating 
form.  Again,  ratings  can  be  done  as  deficiencies 
occur  or  during  the  time  window.  While  this  method 
is  also  satisfactory,  it  reduces  the  amount  of  data 
for  feedback  to  the  G 2  and  commander. 

Rating  both  important  information  items  and 
deficient  item.n.  This  is  a  combination  of  the 
above.  Once  again,  this  method  offers  less 
feedback  data  than  the  suggested  method. 

The  seriousness  of  the  consequences  of  not  rating  all  the 
information  items  must  be  considered.  While  the  options  permit 
diagnosis,  they  severely  limit  the  feedback  that  can  be  given  the 
commander  and  G2.  In  the  case  of  rating  deficient  items  only, 
the  feedback  can  only  be  negative  since  there  will  be  no  data  on 
the  non-rated  items.  In  the  other  cases,  it  will  be  difficult  to 
track  improvement  in  performance  except  for  a  limited  number  of 
items.  As  a  result,  the  entire  picture  may  not  be  presented. 
Potential  problems  may  be  overlooked  and  preventive  action  not 
taken.  It  is  strongly  recommended  that  the  complete  assessment 
method  be  implemented. 

Rating  all  the  items,  while  time  consuming,  gives  a  full  picture 
for  both  diagnosis  and  feedback. 

E.  SUMMARY. 

The  steps  necessary  to  rate  the  effectiveness  of  Military 
Intelligence  information  production  are  as  follows: 

1.  Review  scales  and  definitions  for  the  scales. 

2.  Rate  all  information  items  using  the  timeliness 
scale. 

3.  Rate  all  information  items  using  the  frequency 
scale. 
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4.  Rate  all  information  items  using  the  operational 
perspective  scale  and  the  sample  operational 
perspective  deficiencies. 

5.  Rate  all  information  items  using  the  clarity  scale 
and  the  sample  clarity  deficiencies. 

6.  Rate  all  information  items  using  the  completeness 
scale  and  the  sample  completeness  deficiencies. 

7.  Reestablish  the  benchmarks. 

8.  Review  the  ratings  for  completeness  and  turn  in  to 
the  team  chief. 

F.  IMPORTANCE  OF  THE  RATING. 

The  ratings  from  the  observer-participants  are  the  basis  of  the 
diagnostic  plan  which  is  used  to  determine  the  cause  of  the 
deficiencies  in  the  intelligence.  The  ratings  also  provide  the 
basis  for  the  feedback  to  the  commander  and  G2.  Thus,  they  are 
the  backbone  of  the  assessment  procedure.  Because  of  their 
importance,  the  ratings  must  be  as  accurate  and  as  comprehensive 
as  possible. 


48 


SECTION  6 


DIAGNOSING  MILITARY  INTELLIGENCE  PERFORMANCE 
ANALYZING  THE  RATINGS 

While  the  ratings  provided  by  observer-participants  form  the 
basis  for  conducting  an  assessment,  common  sense  and  an 
understanding  of  what  the  unit  is  trying  to  achieve  are  the 
critical  elements  of  a  successful  diagnosis.  There  are  two 
phases  of  the  diagnosis.  The  first  phase,  the  analysis, 
includes  combining  the  rating  from  data  provided  by  the  different 
observer-participants  and  analyzing  the  combined  deficiency  data. 
The  second  phase,  the  inquiry,  involves  developing  an  inquiry 
strategy  for  conducting  the  diagnosis,  the  actual  gathering  of 
data,  data  analysis,  and  recording  of  the  findings. 

A.  DEFICIENCY  CONSOLIDATION  WORKSHEET. 

The  analysis  of  the  observer-participant  ratings  must  be  done 
rapidly  so  that  diagnosis  can  be  carried  out  during  the  exercise. 
The  Deficiency  Consolidation  Worksheet  provides  a  means  to 
rapidly  record  data  from  the  various  rating  forms  to  produce  a 
visual  display  of  how  the  deficiencies  cluster. 

B.  DESCRIPTION  OF  THE  DEFICIENCY  CONSOLIDATION  WORKSHEET. 

A  copy  of  the  worksheet  is  in  Figure  14.  Copies  are  also  in 
Appendix  C-l,  along  with  instructions  on  its  use  (C-2) .  There 
are  four  different  worksheets,  one  for  each  of  the  four  major 
information  categories.  The  worksheets  are  in  the  same  format. 

In  the  left  column  are  the  information  items  for  the  information 
category  identified  at  the  top  of  the  column.  Across  the  top  of 
the  matrix  and  identifying  the  remaining  columns  are  the  five 
dimensions  the  items  were  rated  on.  A  cell  is  described  in  terms 
of  both  the  vertical  and  horizontal  dimensions.  For  example, 
the  cell  with  the  A  in  it  will  contain  data  about  the  "terrain 
effect  on  the  EN  operation"  which  was  deficient  on  operational 
perspective. 

C.  STEP  4.  ANALYZING  THE  RATINGS;  CONSOLIDATING  THE  PERFORMANCE 
RATING  FORM  DATA. 

A  deficient  item  is  one  that  falls  below  the  observer- 
participant's  benchmark.  For  example,  if  the  benchmark  for 
operational  perspective  is  3,  any  item  rated  4  or  5  on 
operational  perspective  is  deficient  according  to  that  observer- 
participant.  Therefore,  in  order  to  fill  in  the  worksheet,  the 
benchmark  for  each  dimension  for  each  observer-participant  must 
be  known.  The  benchmarks  are  in  the  left  column  of  the 
Performance  Rating  Form. 
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There  are  many  procedures  for  filling  in  the  worksheet.  The 
procedure  used  will  depend  upon  how  many  observer-participants 
are  involved  in  the  assessment.  If  there  are  many,  say  more  than 
10,  a  mechanical  data  consolidation  is  necessary.  If  there  are 
fewer,  the  diagnostician  might  be  able  to  consolidate  just  by 
looking  over  the  Performance  Rating  Forms. 

The  method  suggested  and  illustrated  is  when  there  are  many 
observer-participants.  However,  the  diagnostician  should  use 
whatever  procedure  he  is  most  comfortable  with  for  the  amount  of 
data  that  must  be  consolidated. 

D.  SUGGESTED  PROCEDURE  FOR  COMPLETING  THE  WORKSHEET. 

1.  On  each  Performance  Rating  Form,  circle  the 
deficient  ratings.  For  example,  if  the  benchmark 
for  timeliness  is  3,  circle  all  the  4s  and  5s  in 
the  timeliness  column.  If  the  benchmark  for 
frequency  is  2,  circle  all  the  3s,  4s,  and  5s  in 
the  frequency  column.  Do  the  same  for  the  other 
dimensions. 

2.  When  all  the  rating  forms  have  been  completed, 
place  the  deficient  items  in  the  appropriate  cell 
of  the  consolidation  worksheet.  Place  a  line 
across  the  circled  number  on  the  rating  form  as  the 
data  is  recorded.  This  should  prevent  recording 
the  same  data  more  than  once. 

3.  Record  in  the  appropriate  cell  of  the  consolidation 
worksheet  the  position  that  did  the  rating,  for  example 
G2  OPS.  For  the  timeliness  dimension,  also 

indicate  if  the  item  was  not  received.  When  the 
deficient  items  from  an  observer-participant  have 
been  recorded,  a  check  should  be  placed  at  the  top  of 
the  rating  form  to  indicate  the  data  has  been 
recorded . 

4.  Record  the  deficiencies  for  all  the  rating  forms. 

EXAMPLE  The  example  uses  a  rating  form  for  one  observer- 

participant  (Figure  15)  and  shows  one  of  the 
four  completed  worksheets  for  that  observer- 
participant  (Figure  16) .  The  example 
illustrates  how  the  data  from  the  rating  form 
fell  into  the  different  cells  of  the 
consolidation  worksheet.  It  shows  that  there 
are  15  deficiencies.  Note  that  there  are 
information  items  on  the  Performance  Rating  Form 
which  have  letters  in  the  deficiency  column,  but 
don't  have  circled  ratings.  While  that  item  may 
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have  had  a  deficiency,  the  observer-participant 
still  rated  it  as  acceptable. 

AN  ITEM  TO  BE  DEFICIENT  MUST  BE  RATED  BELOW  THE 
BENCHMARK. 

The  worksheet  (Figure  16)  shows  that  the  G3 
plans  identified  seven  timeliness,  four 
frequency,  one  operational  perspective,  and 
three  clarity  deficiencies.  Note  also  the 
short  hand  for  the  position,  G3P.  The  short 
hand  is  whatever  the  person  filling  in  the  form 
wants  it  to  be,  but  anyone  who  has  to  use  the 
worksheet  later  needs  to  know  how  to  interpret 
the  short  hand. 

EXAMPLE  Figure  17  shows  a  Deficiency  Consolidation 

Worksheet  completed  for  five  different  observer- 
participants  from:  FSE,  G3  OPS,  G3  plans,  ADA, 
and  G2  OPS.  For  example,  it  shows  13  timeliness 
deficiencies,  10  because  the  item  was  not 
received  (circled  and  labeled  A) . 

The  example  represents  the  consolidation  of  one 
rating  session  having  observer-participants  for 
five  positions.  If  there  was  more  than  one 
person  rating  from  the  same  position,  each 
identifying  the  same  deficiency,  then  each 
deficiency  must  be  recorded. 

If  it  is  necessary  to  consolidate  the 
deficiencies  over  several  rating  periods,  a 
combined  consolidation  worksheet  can  be  made. 

It  is  done  by  recording  the  number  of  times  the 
item  was  rated  deficient  for  each  position. 

Thus,  an  entry,  G30-nr(6)  in  the  objectives- 
timeliness  cell  would  mean  that  over  the  course 
of  the  exercise  that  item  was  rated  as  not 
received  by  the  G3  operations  on  six  ratings. 

E.  STEP  4.  ANALYZING.  THE_ RATINGS:  ANALYZING  OF  THE  DEFICIENCY 
CONSOLIDATION  WORKSHEET. 

The  worksheet  presents  a  clustering  of  the  deficiencies.  From 
the  clustering,  the  diagnostician  can  determine  deficiencies  and 
their  priorities  for  diagnosing.  However,  the  most  important 
criteria  for  determining  the  diagnostic  priorities  is  the 
objective  of  the  exercise. 
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If  the  exercise  has  a  limited  training 
objective,  such  as  to  train  intelligence  to 
rapidly  analyze  and  produce  targeting  data,  the 
diagnostician  may  want  to  focus  on  deficient 
items  related  to  enemy  unit  locations  and 
critical  nodes/high  payoff  targets. 

If  the  training  objective  is  the  rapid  handling 
of  all  intelligence  information,  the 
diagnostician  may  want  to  concentrate  on 
timeliness  deficiencies. 

If  the  assessment  environment  is  a  full  division 
exercise  where  a  full  range  of  MI  functions  are 
exercised,  the  diagnostician  may  prioritize 
deficiencies  based  on  their  potential  effect  on 
the  outcome  of  the  exercise. 

If  in  doubt,  the  diagnostician  may  prioritize 
based  on  the  assigned  weights  of  the  deficient 
items.  A  general  rule  is  that  it  is  more 
important  to  diagnose  items  that  observer- 
participants  place  a  high  weight  on  than  an  item 
that  was  assigned  a  low  weight. 

P.  STEP  5.  DETERMINING  WHERE  DEFICIENCIES  OCCUR:  INTERPRETING 
THE  WORKSHEET. 

Each  cell  represents  a  cluster  of  deficiencies.  Reading  down  the 
deficiency  column  indicates  how  often  the  specific  deficiency 
occurred.  Reading  across  item  row  indicates  how  often  that 
specific  item  was  deficient.  Looking  within  the  cells  indicates 
who  (the  position)  was  having  the  problems. 

EXAMPLE  An  example  analyses  of  the  information  on  the 

Deficiency  Consolidation  Worksheet  is  in  Figure 
18.  There  were  five  observer-participants,  one 
each  from:  G3  plans,  G3  operations,  FSE,  G2 
operations,  and  ADA.  What  stands  out  are  the 
four  items,  labeled  A,  that  neither  the  G3  OPS 
or  G3  plans  received.  Also,  timeliness  errors 
predominate  (there  are  13  deficiencies  listed  in 
the  timeliness  column  as  opposed  to  nine  in  the 
clarity  column) .  The  G3  plans  is  identifying 
the  most  deficiencies  (the  circled  data  in  the 
cells) ,  and  clarity  deficiencies  seem  to  stem 
from  poor  organization  (note  the  A  by  all  but 
one  of  the  deficiencies  in  the  clarity  column. 
The  A  from  the  sample  deficiency  list  for 
clarity  was  poor  organization) . 


63 


DEFICIENCY  CONSOLIDATION  WORKSHEET 


This  example  is  for  one  of  the  consolidation 
worksheets.  The  worksheets  for  the  other 
information  items  would  be  included  in  an  actual 
interpretation . 

In  this  example,  item  weights  are  not 
considered.  If  they  were  to  be  considered,  a 
different  format  for  consolidation  would  have  to 
be  used  or  it  would  be  necessary  to  concentrate 
on  a  particular  observer-participant.  In  the 
latter  case,  the  information  requirements 
profile  would  be  used  along  with  the  Performance 
Rating  Form  and  worksheet. 
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SECTION  7 


DIAGNOSING  MILITARY  INTELLIGENCE  PERFORMANCE: 

THE  INQUIRY  STRATEGY 

During  this  step,  the  plan  for  collecting  the  data  necessary  to 
determine  the  reasons  for  the  deficiencies  is  laid  out.  It 
includes  determining  where  to  go  to  collect  the  data, 
determining  the  questions  the  diagnostician  needs  to  answer, 
determining  the  best  way  to  get  the  answers,  and  recording  the 
data  after  the  inquiry. 

A.  THE  PLANNED  INQUIRY  GUIDE. 

The  Planned  Inquiry  Guide  is  shown  in  Figure  19.  There  is 
another  copy  in  Appendix  C-3,  along  with  instructions  on  its  use. 
The  inquiry  guide  helps  to  organize  the  inquiry  strategy. 

B.  DESCRIPTION  OF  THE  PLANNED  INQUIRY  GUIDE. 

The  inquiry  guide  has  four  vertical  columns,  "deficiency  or  areas 
to  diagnose,"  "potential  causes,"  "where  to  start,"  and 
"questions  to  be  answered."  The  columns  of  the  guide  have  been 
left  open  to  indicate  that  it  may  take  more  than  one  page  to  plan 
the  inquiry.  How  many  it  takes,  or  how  many  deficiencies  go  on  a 
page  depends  entirely  on  the  diagnostician,  how  big  he  writes, 
how  many  deficiencies  there  are,  and  how  he  wants  to  organize  the 
inquiry. 

C.  STEP  6.  DEVELOP  A  DIAGNOSTIC  PLAN:  USING  THE  INQUIRY  GUIDE. 

1.  Deficiency  or  Areas  to  Diagnose. 

In  the  first  column,  the  diagnostician  should  put  the 
deficiency  areas  that  he  wants  to  diagnose.  These  were 
determined  from  prioritization  of  information 
from  the  Deficiency  Consolidation  Worksheet. 

2.  Potential  Causes. 

Filling  in  this  column  is  at  the  discretion  of  the 
diagnostician.  Because  of  the  experience  of  the 
diagnostician,  he  may  have  some  ideas  on  why  the 
deficiencies  occurred.  He  may  want  to  note  in  this 
column  some  things  to  check  out. 

3.  Where  to  Start. 

The  inquiry  must  begin  somewhere.  The  types  of 
deficiencies  and  the  experience  of  the  diagnostician  are 
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PLANNED  INQUIRY  GUIDE 


the  major  factors  in  defining  a  place  to  start  seeking 
answers.  The  Information  Item  Deficiency  to  Source 
Matrix  in  Figure  20  and  in  Appendix  C-5,  is  an  aid  to 
determining  where  to  go  to  begin  the  inquiry. 

Information  Item  Deficiency  to  Source  Matrix: 

In  the  left  column  are  the  major  categories  of 
information  items.  Across  the  top  are  the  types  of 
deficiencies.  Within  the  cell  is  the  recommended  place 
to  begin  the  inquiry. 

EXAMPLE  If  possible  courses  of  enemy  action  had  not  been 

received,  the  place  to  begin  inquiry  is  the  all 
source  production  section. 

If  weather  information  was  deficient  because  of 
lack  of  completeness,  the  place  to  begin  inquiry 
is  G2  operations. 

4.  Questions  which  Must  be  Answered. 

In  order  to  determine  why  the  deficiencies  occurred,  the 
diagnostician  must  find  the  answers  to  questions  based  on 
the  observed  information  deficiencies.  The  Diagnostic 
Question  Inventory  and  its  flow  charts  are  aids  to  help 
determine  the  most  appropriate  questions  to  ask.  The 
aids,  with  a  complete  set  of  directions  on  how  to  use 
them,  are  in  Appendix  C-6,  C-7  and  C-8.  Using  the 
diagnostic  question  inventory,  the  diagnostician 
determines  the  questions  he  wants  answered.  They  are 
recorded  in  the  "questions  which  must  be  answered"  column 
of  the  inquiry  guide. 

D.  DIAGNOSTIC  QUESTION  INVENTORY. 

The  question  inventory  contains  sequenced  questions,  organized  by 
type  of  deficiency.  An  example  from  the  inventory  is  shown  in 
Figure  21.  The  question  inventory  should  be  reviewed  to  ensure 
that  all  the  questions,  relevant  to  the  particular  deficiency 
being  diagnosed,  are  considered.  The  question  inventory  is  not 
comprehensive  and  only  serves  as  a  guide.  It  may  be  supplemented 
by  questions  or  areas  of  inquiry  that  experience  indicates  are 
important  and  related  to  the  deficiency. 
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INFORMATION  ITEM  DEFICIENCY  TO  SOURCE  MATRIX 


Information  Item 

Category 

Items  Omitted 

Poor  Quality  Items 

Timeliness/ 

Frequency 

Operational 

Perspective 

BATTELFIELD  AREA 

G2 

ASPS  G2  Ops 

Weather 

USAF  WO 

G2 

G2  Ops  G2  Ops 

Terrain 

ENG  TM 

G2 

ASPS 

Area  conditions 

ASPS 

G2 

ASPS 

ENSIT 

G2  Ops 

G2 

G2  Ops 

Disposition  &  Comp. 

G2  Ops/CM&D 

G2 

G2  Ops 

Strength 

G2  Ops/CM&D 

G2 

G2  Ops 

Activities 

G2  Ops/CM&D 

G2 

G2  Ops 

ENCOAS 

ASPS 

G2 

ASPS 

Possible  COAs 

ASPS 

G2 

ASPS 

Other  Info  on  Prob.  COAs 

ASPS/CM&D 

G2 

ASPS 

OPSEC 

G2/CI  TEAM 

G2 

G2  Ops 

EN  Recce/Intel 

ASPS/CM&D 

G2 

ASPS/G2  Ops 

EN  REC 

ASPS/CM&D 

G2 

ASPS/G2  Ops 

EN  Special  Ops 

ASPS/CM&D 

G2 

ASPS/G2  Ops 

FR  Vulnerabilities 

G2/ASPS 

G2 

ASPS/G2  Ops 

Deception 

G2/ASPS 

G2 

ASPS/G2  Ops 

UNIT  SPECIFIC  COMMENTS: 


Figure  20.  The  Information  Item  Deficiency  to  Source  Matrix  Form. 
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DEFICIENCY  Dimension  1:  TIMELINESS 

Timeliness  is  unusual  in  that  if  you  find  a  possible  cause  for  a 
deficiency,  correcting  it  might  not  provide  enough  additional 
time  to  have  prevented  the  deficiency.  You  may  be  dealing  with 
multiple  causes,  which  together  contributed  to  the  deficiency. 

As  a  result,  you  will  not  always  stop  asking  questions  when  you 
have  discovered  a  cause.  Follow  questioning  in  sequence  unless 
you  are  to  go  to  a  different  question. 

The  first  set  of  questions  is  to  determine  whether  something  went 
wrong  with  dissemination.  In  other  words,  something  got  fouled 
up  getting  the  information  out  to  the  observer-participant.  Stop 
asking  questions  about  an  information  item  when  you  have  fully 
identified  what  went  wrong.  Go  through  each  information  item  on 
your  data  sheet.  It  may  become  apparent  that  the  same  problem  is 
responsible  for  a  deficiency  in  many  items.  Use  common  sense  on 
when  to  stop  asking  questions.  Item  la  below  is  a  good  example, 
if  there  are  no  procedures  for  prioritizing  information,  that  may 
affect  all  deficient  items,  but  that  doesn't  mean  questions  Id 
and  le  should  not  be  asked  since  they  have  nothing  to  do  with 
prioritization. 

Questions  1  and  2  are  concerned  with  the  dissemination  of 
information  from  the  MI  producer  to  the  user  without  respect  to 
form;  it  makes  no  difference  if  dissemination  is  a  formal 
briefing,  phone  call,  or  radio  transmission. 

1.  Was  information  item  sent  to  the  user?  If  no,  go  to  Question 

2.  If  yes,  then: 

a.  Was  a  priority  placed  on  getting  the  information  sent? 

If  no,  check  to  see  if  there  were  adequate  procedures  for 
prioritizing  communications.  If  there  weren't,  maybe 
there  should  be.  If  there  were,  why  weren't  they 
followed? 

b.  Did  the  priority  cause  other  information  to  be  sent 
first,  thus  delaying  dissemination?  If  yes,  was 
information  prioritized  correctly,  were  procedures 
followed,  or  was  everything  done  correctly,  but  more 
important  information  needed  to  be  sent  out  first? 

c.  Did  the  person  responsible  for  dissemination  respond  to 
the  priority?  If  no,  why  not?  If  yes,  the  problem  may 
be  identified  by  questions  Id  or  le. 


Figure  21.  Example  questions  from  the  diagnostic  question 
inventory  on  timeliness. 
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Figure  21.  (Cont) 

d.  Did  the  person  responsible  for  dissemination  select  the 
best  means/alternate  means  to  send  the  information?  If 
no,  look  again  at  procedures  to  determine  if  they  exist, 
cover  the  situation,  and  were  followed. 

e.  Was  it  confirmed  that  the  user  received  the  information? 
If  no,  did  procedures  exist,  did  they  cover  the 
situation,  and  were  they  followed?  Could  confirmation  be 
carried  out? 

2.  If  you  identified  a  possible  cause  for  timeliness  deficiency, 
would  correcting  it  have  resulted  in  enough  time  being  saved  to 
have  prevented  the  deficiency?  If  yes,  have  you  identified  the 
entire  cause? 
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In  addition  to  the  question,  the  diagnostician  should  consider 
how  he  wants  to  find  the  answer.  There  is  considerable 
flexibility  in  obtaining  answers  to  the  questions.  The 
diagnostician  may  conduct  interviews  with  individuals  in  the  MI 
production  system,  observe  procedures  and  production  flow,  review 
documents,  or  any  combination  of  these.  The  questions  are  those 
the  diagnostician  must  answer.  The  diagnostician  continues 
answering  questions  until  he  determines  what  could  have  caused 
the  deficiency. 

EXAMPLE  A  completed  Planned  Inquiry  Guide  is  shown  in 

Figure  22.  Three  different  deficiencies  are 
addressed.  Note  that  included  in  the  right 
column  is  where  to  go  based  on  the  answer  to  the 
question.  The  questions  in  this  column  are 
derived  from  the  Diagnostic  Question  Inventory 
in  Appendix  C.  Note  also  that  column  2 
reflects  the  unique  unit  and  situational 
knowledge  of  the  diagnostician. 

E.  STEP  7.  DIAGNOSING  THE  INTELLIGENCE  INFORMATION  PRODUCTION 
SYSTEM. 

Using  the  Planned  Inquiry  Guide,  the  diagnostician  moves  through 
the  intelligence  production  system,  acquiring  data  through 
observation,  interviews  with  intelligence  producers,  and  reviews 
of  documents.  The  inquiry  should  result  in  a  series  of  chained 
events  triggered  by  responses  to  the  questions  started  from  the 
Planned  Inquiry  Guide. 

EXAMPLE  If  the  diagnostician  is  working  to  determine  the 

cause  of  an  omission  deficiency  reported  by  a  G3 
observer-participant,  the  inquiry  may  have  begun 
with  a  review  of  the  PIR/IR  listing  in  the  CM&D. 
Upon  determining  through  observation  that  G3 
information  items  were  included  in  the 
collection  plan,  the  diagnostician  must  move  to 
other  segments  of  the  production  system  to 
determine  if  the  information  was  received, 
processed,  and  dispatched  to  the  G3. 

While  appearing  straight  forward,  the  inquiry  process  is 
detailed  and  complex.  The  diagnostician  must  be  as  rigorous  as 
possible  given  the  available  time  he  has  to  conduct  the  inquiry. 

F.  RECORDING  THE  RESULTS. 

Diagnostic  Record:  Figure  23  is  a  completed  Diagnostic 
Record.  A  blank  copy  is  provided  in  Appendix  C-9,  along  with 
directions  for  its  use  (Appendix  C-9) .  The  format  parallels 
the  Planned  Inquiry  Guide.  Although  the  example  is  for  three 
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I  ne  who  knows  AIM  best. 


deficiencies,  how  many  pages  or  how  many  deficiencies  should  go 
on  a  page  depends  on  the  amount  of  information  and  how  the 
diagnostician  organizes  it.  The  form  has  been  left  open  to 
provide  more  space  for  recording  data  and  making  notes  when 
preparing  feedback.  The  purpose  of  the  Diagnostic  Record  is  to 
help  organize  the  findings.  It  is  a  tool  to  help  prepare  for 
giving  feedback,  not  a  document  that  is  handed  out  as  part  of  the 
feedback. 

Inquiry  Data:  In  this  column  is  recorded  all  the  data  gathered 
to  support  the  identification  of  the  cause (s)  of  the  deficiency. 

Problems  Identified:  This  column  has  a  description  of  the 
problem (s)  discovered. 

Related  Information  Deficiency:  This  column  is  a 
restatement  of  the  deficiencies  that  resulted  from  the 
problem. 

Causes  of  Deficiencies:  The  conclusions  drawn  from  the  data 
in  the  first  column  are  recorded  here.  This  column 
indicates  why  the  deficiency  occurred. 

Recommended  Solutions:  This  column  does  not  have  to  be 
filled  out.  It  can  be  if  the  diagnostician  believes  that  he 
may  have  an  acceptable  solution.  However,  it  is  not  the 
purpose  of  the  diagnosis  within  this  assessment  procedure  to 
advise  on  remedy. 

G.  ADMINISTRATIVE  CONSIDERATIONS. 

The  planning  and  conducting  of  the  inquiry  into  causes  of 
information  deficiencies  takes  time  and  is  done  while  the 
exercise  is  in  progress.  It  must  be  done  with  tact  and  without 
disruption  to  the  exercise.  Because  the  diagnostics  is  so  time 
constrained,  only  the  most  critical  deficiencies  should  be 
diagnosed  during  the  exercise. 

NOTE:  A  diagnosis  could  be  done  after  the  exercise,  but  it  would 
have  to  be  modified  some.  Procedures  couldn't  be  observed,  some 
of  the  staff  might  be  difficult  to  get  to  to  interview,  etc. 

It  cannot  be  stressed  enough  that  a  successful  diagnosis  is  based 
on  the  common  sense  and  understanding  of  the  unit  of  the 
experienced  person  conducting  the  diagnosis. 

H.  SUMMARY- 

The  steps  in  diagnosing  Military  Intelligence  performance  are  as 
follows: 
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1.  Consolidate  the  performance  rating  data  using  the 
Deficiency  Consolidation  Worksheet. 

2.  Analyze  the  consolidation  worksheets  by  identifying 
clusters  of  performance  deficiencies  or  deficiencies  of 
high  weighted  information  items. 

3.  Plan  the  inquiry  strategy  using  the  Planned  Inquiry 
Guide. 

4.  Gather  data  to  determine  the  causes  for  the 
deficiencies. 

5.  Record  the  data  and  the  conclusion  based  on  that  data  on 
the  Diagnostic  Record. 
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SECTION  8 


PROVIDING  FEEDBACK 

The  purpose  for  conducting  the  assessment  is  to  provide  feedback 
to  the  commander  and  the  G2  on  how  well  Military  Intelligence  met 
the  needs  of  the  command  during  the  designated  exercise.  While 
the  commander  and  the  G2  should  determine  how  frequently  they 
want  to  receive  feedback,  the  team  chief  and  diagnostician  need 
to  work  together  to  prepare  and  provide  the  feedback. 

Unless  directed  by  the  commander  or  G2  to  only  provide  specific 
feedback,  the  content  of  the  feedback  should  be  determined  by 
when  the  presentation  of  the  feedback  is  scheduled.  Feedback  can 
be  given  periodically  during  the  exercise  and  at  the  end  of  the 
exercise. 

FEEDBACK  SHOULD  ALWAYS  BE  TO  THE  G2  AND  TO  THE  COMMANDER  FROM  THE 
G2 . 


A.  PERIODIC  FEEDBACK  DURING  THE  EXERCISE. 

The  periodic  feedback  should  not  be  a  presentation  of  a 
statistical  analysis  of  the  Performance  Rating  Forms. 

Just  as  the  diagnosis  was  based  on  common  sense  and  an 
understanding  of  the  unit,  providing  feedback  during  the 
exercise  requires  the  same  traits.  Since  time  is  limited  for 
preparing  as  well  as  presenting,  the  feedback  should  be  limited 
to  major  findings. 

The  best  source  of  data  for  preparing  feedback  is  the  Deficiency 
Consolidation  Worksheets  which  provide  clustered  information. 

The  most  important  periodic  feedback  should  concern  the 
deficiencies  that  the  diagnostician  selected  for  inquiry  and  the 
findings  from  the  inquiry. 

The  weights  of  the  information  items  which  were  deficient  might 
also  be  of  value  in  the  periodic  feedback.  This  information  can 
be  obtained  from  the  observer-participants'  Information 
Requirements  Profile  (items  weighted  100). 

Finally,  if  there  has  been  more  than  one  rating  period,  or  more 
than  one  feedback  session,  changes  in  performance,  either 
positive  or  negative  will  be  good  feedback.  This  kind  of  trend 
information  can  be  obtained  by  comparing  changes  showing  up  in 
the  consolidation  worksheets. 
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B.  POST-EXERCISE  FEEDBACK. 


The  post-exercise  feedback  must  include  the  information  listed 
above  for  the  periodic  feedback.  That  information  is  essential. 
If  time  is  limited,  it  is  the  highest  priority  feedback.  The 
commander  and  the  G2  must  at  least  know  what  went  wrong  where  and 
why,  so  they  can  implement  training  or  operational  fixes.  Time 
permitting,  however,  the  Performance  Rating  Forms,  used  in 
conjunction  with  the  Information  Requirements  Profile,  can 
provide  a  wealth  of  quantitative  data.  Because  there  is 
potentially  so  much  information,  it  may  be  necessary  to  combine 
it  in  order  to  present  a  structured  picture  of  the  assessment 
results.  There  are  many  ways  the  data  can  be  analyzed, 
therefore,  only  a  few  are  presented.  The  following  are  some 
useful  techniques  for  consolidating  the  data: 

Consolidating  across  deficiency  dimensions:  Never  combine 
the  data  for  deficiency  dimensions:  timeliness,  frequency, 
operational  perspective,  clarity,  and  completeness  should 
never  be  combined.  Such  a  consolidation  would  not  be 
meaningful  since  important  information  would  be  hidden. 

Consolidating  deficiencies  by  observer-participants:  Very 
important  feedback  can  be  provide  by  consolidation  of  data 
within  an  observer-participant  group.  For  example,  if  there 
are  several  representatives  from  the  G3  section  it  is 
all  right  to  combine  their  data.  It  is  not  acceptable  to 
combine  data  from  the  G3  with  the  FSE.  When  combining  the 
deficiency  data,  all  that  is  needed  is  a  total  count  for 
certain  types  of  deficiencies.  For  example,  the  G 3  section 
for  Enemy  Courses  of  Action  rated  the  information  item 
"objective"  deficient  on  6  occasions  during  the  exercise. 

Feedback  should  be  provided  in  terms  of  each  observer- 
participant  group. 

Consolidating  across  rating  periods:  This  could  be  done  for 
each  observer-participant  for  any  deficiency  dimension  if  it 
is  desired  to  present  a  total  "score"  or  a  big  picture  of 
what  happened  during  the  exercise.  However,  it  is  best  not 
to  present  the  total  "score"  alone.  It  should  be  presented 
along  with  the  data  for  the  rating  periods,  so  that  the 
audience  has  the  opportunity  to  see  what  makes  up  the  total 
score  and  what  trends  are  revealed. 

C.  ANALYZING  THE  DEFICIENCY .CONSOLIDATION  WORKSHEET. 

The  feedback  from  the  analysis  should  not  be  "one  measure,"  but 
rather  a  pattern  of  "measures."  There  are  four  measures  we 
propose  (details  for  computing  them  are  in  Appendix  D) : 
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Comprehensiveness  indicates  the  number  of  information 
items  which  were  acceptable.  For  example,  if  one  item 
out  of  the  58  was  rated  deficient,  then  57  would  be 
acceptable.  The  comprehensiveness  score  is  98%  (57/58  x 
100) .  The  higher  the  percent,  the  better  the 
performance. 

Criticality  indicates  the  importance  of  the 
information  items  which  were  missed.  It  is  based 
on  the  weight  given  the  items  by  the  observer- 
participant.  The  higher  the  number,  the  worse  the 
performance.  For  example,  if  only  one  item  was 
missed,  but  it  was  a  100  weighted  item,  a  critical 
item  was  not  acceptable. 

Effectiveness  represents  the  overall  usefulness  of 
the  information  to  the  observer-participant.  It  is 
based  on  the  number  of  information  items  which  were 
not  deficient  and  the  weights  of  the  items.  It  is 
expressed  as  a  percent.  The  higher  the  percent, 
the  better  the  performance. 

Distribution  of  performance  ratings  represent  how 
the  observer-participant  rated  each  dimension 
(timeliness,  frequency,  etc.)  on  the  scale.  If  a 
benchmark  had  been  between  3  and  4,  for  example  and 
the  rating  showed  all  Is,  then  performance  in  that 
dimension  would  have  been  as  good  as  it  could  get. 

The  four  measures  can  be  presented  in  many  different  ways.  The 
first  three  could  be  shown  for  the  deficiency  dimensions 
combined,  or  for  each  deficiency  dimension.  They  could  be  shown 
for  the  exercise,  or  broken  out  by  rating  periods  to  show  trends. 
The  distribution  of  ratings  can  be  shown  for  the  exercise  or 
broken  out  by  rating  periods  to  show  trends. 

Trends  in  Performance  as  Feedback:  Providing  the  commander 
and  G2  feedback  on  the  trends  in  MI  performance  is  next  in 
importance  to  the  what,  where,  why  of  the  deficiencies.  The 
trends  place  performance  in  the  time  perspective  of  the 
exercise  events.  With  the  trend  data,  the  commander  is 
better  able  to  judge,  for  example,  the  impact  on  intelligence 
performance  of  a  jump  or  communications  breakdown. 

Positive  and  Negative  Feedback:  The  purpose  of  having  the 
five  point  rating  scales  is  to  have  a  means  of  getting 
positive  information  as  well  as  negative.  It  is  critical 
that  positive  as  well  as  negative  feedback  be  provided. 

Often  the  reasons  for  doing  something  well  can  provide 
solutions  for  improving  inadequate  performance. 
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Conflicting  Data:  It  would  not  be  unusual  to  discover  that 
the  observer-participant  rated  information  deficient  and  the 
diagnostician  couldn't  find  any  deficiency  in  MI  information 
production  performance.  While  it  may  be  difficult  to  explain 
the  conflicting  data,  it  should  at  least  be  brought  up  for 
consideration.  (If  it  is  important  to  resolve  the  conflict, 
so  that  direction  can  be  given) . 

Comparing  Performance  Across  MI  Units:  The  methodology 
presented  for  assessing  MI  information  processing 
effectiveness  has  one  purpose,  to  assist  MI  in  improving 
their  performance.  It  is  not  a  grade,  nor  should  it  be  used 
as  such.  Exercises  operate  with  different  objectives  and 
under  different  situational  constraints.  This  requires  the 
assessment  results  to  be  interrupted  within  the  constraints 
of  the  specific  exercise. 
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SUPPLEMENTAL  MATERIALS  FOR  IDENTIFYING 
AND  PRIORITIZING  INFORMATION  REQUIREMENTS 

A-l  Blank  Copy  of  the  Information  Requirements  Profile 

A-2  Instructions  for  Weighing  the  Information  Hierarchy 

A- 3  Definitions  of  the  Information  Items  in  the  Information 
Hierarchy 

A- 4  Timeliness  Dimension  Scale 

A-5  Frequency  Dimension  Scale 

A-6  Operational  Perspective  Dimension  Scale 

A- 7  Clarity  Dimension  Scale 

A-8  Completeness  Dimension  Scale 

A-9  Instructions  for  Setting  Benchmarks 


A-l 


-V 


APPENDIX  A-l 

BLANK  COPY  OF  THE  INFORMATION  REQUIREMENTS  PROFILE 
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INFORMATION  REQUIREMENTS  PROFILE 


(INSTRUCTIONS: 


1 .  Assign  zero  10  any  Information  Hems  you  do  not  want 

2.  Assign  1 00  to  your  most  Important  Information  hams. 

3.  Assign  a  number  between  0  and  1 00  to  remaining  Hams 
to  reflect  the  relative  importance  of  the  Ham. 

4.  List  specific  data  items  you  want  emphasized  under 
special  notes 


NAME _ 

POSITION . 
DATE _ 


IMPORTANCE  MOW  , 

SPECIAL  NOTES  l 

APPENDIX  A-2 


INSTRUCTIONS  FOR  WEIGHING  THE 
INFORMATION  HIERARCHY 


The  weighing  of  the  items  in  the  information  hierarchy  results 
in  the  prioritization  of  your  information  requirements.  A  copy 
of  the  Information  Requirements  Profile  form  and  the  definitions 
of  elements  in  the  information  hierarchy  are  necessary  in  order 
to  complete  the  weighing. 


Step  1.  Review  the  information  items  in  the  left  column  and 

their  definitions. 


Step  2. 


Step  3. 


NOTE: 


Step  4. 


Step  5. 
Step  6. 


Based  on  the  scoring  period  previously  defined, 
identify  the  information  items  that  are  most 
important  for  you  to  have.  Give  those  items  100 
points  by  placing  100  in  the  importance  weight 
column  in  space  by  that  item. 

If  there  are  any  information  items  you  do  not  need, 
give  them  a  score  of  0  and  record  in  the 
appropriate  space. 

There  is  no  limit  on  the  number  of  information  items 
that  are  most  important  or  are  not  needed.  You  may 
consult  the  definitions  as  often  as  necessary  during 
the  weighing. 

Compare  the  remaining  information  items  with  the 
ones  you  have  given  100  points.  Give  each  item 
points  based  on  how  important  you  think  it  is 
relative  to  the  100  point  item(s) .  For  example,  if 
"main  efforts"  was  given  100  points  and  you  think 
"threat  advance"  is  only  half  as  important,  give  it 
50  points.  You  can  give  an  item  anywhere  from  0  to 
100  points.  Record  the  weights  you  give  for  each 
item  in  the  appropriate  space. 

Make  sure  that  there  is  a  weight  for  each  item. 

Review  the  weights  and  change  any  weights  that  you 
think  may  not  be  appropriate.  Make  sure  your  name, 
position,  and  the  date  are  on  the  information 
requirements  profile  form. 
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APPENDIX  A-3 


INFORMATION  REQUIREMENTS  HIERARCHY 
DEFINITIONS  OF  ELEMENTS 

Definitions  for  information  items  contained  in  the  information 
requirements  hierarchy  are  presented  under  their  area  and  group 
headings.  The  term  information  is  used  instead  of  intelligence 
to  ensure  that  all  possible  output  forms  are  included.  While 
information  such  as  unit  locations  may  be  in  another  broader 
information  area,  the  hierarchy  treats  each  as  a  discrete  output 
of  intelligence  production.  Therefore,  information  items  at  all 
levels  are  mutually  exclusive  or  independent  of  each  other,  i.e., 
do  not  assume  that  because  "description  of  enemy  courses  of 
action  might  include  "unit  locations"  is  redundant.  Remember, 
all  information  items  are  independent. 

Level  1:  BATTLEFIELD  AREA 

Specific  aspects  of  the  battlefield  which  includes  the  unit's 
area  of  operation  and  area  of  interest,  are  delineated  into 
three  topical  groups:  weather,  terrain,  and  other  battlefield 
area  conditions. 

Level  2:  WEATHER 

Weather  in  the  battlefield  is  divided  into  three  output  subsets 
of  information: 

Level  3: 

WEATHER  SITUATION.  Current  and  projected  weather  types, 
including,  but  not  limited  to  what  weather  will  occur,  when,  and 
where . 

WEATHER  EFFECTS  ON  EN.  How,  when,  and  where  current  and 
projected  weather  types  will  affect  current  and  projected  enemy 
operations  or  courses  of  action. 

WEATHER  EFFECTS  ON  FR.  How,  when,  and  where  current  and 
projected  weather  types  will  affect  current  and  projected 
friendly  operations  or  courses  of  action. 

Level  2 :  TERRAIN 

Battlefield  terrain  is  divided  into  three  output  subsets  of 
information: 

Level  3: 

TERRAIN  SITUATION,  i.e.,  elevation,  vegetation,  mobility,  and 
soil  conditions. 
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TERRAIN  EFFECTS  ON  EN,  i.e.,  degree  of  slope,  vegetation,  and 
soil  conditions  in  the  battlefield  that  will  shape  enemy 
operations . 

TERRAIN  EFFECTS  ON  FR,  i.e.,  degree  of  slope,  vegetation,  and 
soil  conditions  in  the  battlefield  that  will  shape  friendly 
operations. 

Level  2:  BATTLEFIELD  AREA  CONDITIONS 

Information,  other  than  weather  or  terrain,  pertaining  to 
physical  aspects  of  the  battlefield. 

Level  3: 

EXISTING  BATTLEFIELD  CONDITIONS.  Other  physical  aspects  of  the 
battlefield,  excluding  weather  or  terrain  as  described  above, 
such  as  avenues  of  approach,  status  of  lines  of  communication,  or 
obstacles  within  the  area  of  operations. 

EFFECTS  ON  EN  OPERATIONS.  How,  when,  and  where  other  battlefield 
conditions  will  shape  the  enemy  operations. 

EFFECTS  ON  FR  OPERATIONS.  How,  when,  and  where  other  battlefield 
conditions  will  shape  the  friendly  operations. 

Level  1:  ENEMY  SITUATION 

Current,  dynamic,  and  changing  enemy  situations  within  the  units 
area  of  operation  and  area  of  interest  are  delineated  into  three 
topical  groups:  enemy  disposition  and  composition,  strength  of 
enemy  forces  by  echelon,  and  recent  and  present  significant 
activities. 

Level  2:  EN  DISPOSITION  AND  COMPOSITION 

Organization  for  combat  and  current  deployment  pattern  of  enemy 
elements  is  further  subdivided  into  ten  information  items: 

Level  3: 

FORWARD  TRACE.  Portrayal  of  actual  linear  deployment  of  the  most 
forward  enemy  forces  in  contact  within  the  area  of  operations. 

UNIT  LOCATIONS.  Identifies  location  using  point,  center  of  mass, 
or  area  for  all  enemy  units  in  the  area  of  interest. 

MAIN  EFFORTS.  Enemy  tactical  efforts  in  the  area  of  operations 
in  which  he  has  committed  significant  portions  of  available 
combat  power,  including  location,  disposition,  and  composition. 
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COMBAT  SUPPORT.  Disposition,  composition,  and  location  of  enemy 
combat  support  elements  in  the  area  of  interest. 

ECHELONMENT.  Disposition,  composition,  subordination,  and 
location  of  related  enemy  echelons,  i.e.,  divisions,  fronts  and 
armies,  in  the  area  of  interest. 

RESERVES.  Disposition,  composition,  and  location  of  uncommitted 
enemy  combat  units  in  the  area  of  interest. 

STAGING  AREAS.  Information  on  location  of  enemy  assembly/ staging 
areas,  either  in  use  or  available,  in  the  area  of  interest. 

COMBAT  SERVICE  SUPPORT.  Disposition,  composition,  and  location 
of  enemy  combat  service  support  elements  in  the  area  of 
interest . 

AIR  FORCES.  Disposition,  composition,  and  location  of  enemy  air 
forces  in  the  area  of  interest. 

C2 .  Disposition,  composition,  and  location  of  enemy  command  and 
control  elements  in  the  area  of  interest. 

Level  2:  STRENGTH  OF  EN  FORCES  BY  ECHELON 

Numerical  status  or  combat  readiness  of  enemy  units  based  on 
attrition  in  the  area  of  interest,  by  specific  echelons  is 
divided  into  five  information  items: 

Level  3: 

READINESS  BY  ECHELON.  Available  combat  potential  of  enemy  units 
in  the  area  of  interest. 

SUPPLY  STATUS/RATES  BY  ECHELON.  Enemy  resupply  status/ rates  for 
items  such  as  fuel  or  ammunition  in  the  area  of  interest. 

ENEMY  CRITICAL  NODES/HPT  (HIGH  PAYOFF  TARGETS) .  Formations, 
locations,  or  facilities  whose  capability,  strength,  or  presence 
is  pivotal  to  continuation  of  enemy  operations  within  the  area  of 
interest  and  whose  destruction  or  disruption  provides  advantage 
to  friendly  forces. 

LEVEL  OF  EN  MORALE.  Morale,  well-being,  and  willingness  to  fight 
of  enemy  units  which  effects  their  capability  in  the  area  of 
interest . 

STRENGTH  OF  AIR  FORCES.  Strength/capability  of  enemy  air  forces 
in  the  area  of  interest. 

NBC.  Strength/capabilities  of  enemy  nuclear,  biological,  and 
chemical  elements  in  the  area  of  interest. 
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Level  2:  RECENT/ PRESENT  SIGNIFICANT  ACTIVITIES 

Recent  (up  to  36  hours  old)  and  present  significant  activities 
are  divided  into  six  information  items: 

Level  3: 

COMBAT  ACTION.  Enemy  combat  activities,  including  but  not 
limited  to  unusual  aspects  such  as  actions  involving  nuclear, 
biological,  or  chemical  combat  in  the  area  of  interest. 

C2  ACTIVITY.  Enemy  command  and  control  activities,  i.e., 
communication  activity  or  command  post  relocations,  in  the  area 
of  interest. 

SUSTAINMENT.  Enemy  sustainment  activities,  i.e.,  resupplying  or 
repairing,  in  the  area  of  interest. 

INTELLIGENCE  ACTIVITIES.  Enemy  intelligence  activities,  i.e., 
reconnaissance  patrolling,  radio  intercept,  or  other  covert  or 
overt  activities,  in  the  area  of  interest. 

Level  1:  ENEMY  COURSES  OF  ACTION 

Enemy  courses  of  action  (intentions)  that  are  likely  or  possible 
are  delineated  into  two  topical  groups:  enumerate  possible  enemy 
courses  of  action  and  other  information  for  probable  enemy 
courses  of  action. 

Level  2:  ENUMERATE  POSSIBLE  ECOAs 

Possible  enemy  courses  of  action  are  divided  into  the  following 
information  items: 

Level  3: 

MISSION.  Probable  or  known  specific  tasks/missions  of  related 
enemy  elements  in  the  area  of  interest. 

OBJECTIVES.  Assigned  objectives  of  enemy  units,  whether  force  or 
terrain  oriented,  in  the  area  of  interest. 

FORCES.  Specific  enemy  forces  related  to  specific  courses  of 
action,  including,  but  not  limited  to  their  composition, 
locations,  strength,  and  disposition. 

TERRAIN  CONSIDERATIONS.  Influence  of  terrain  related  to  specific 
enemy  courses  of  action,  i.e.,  routes  of  advance  or  crossing 
sites. 
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ECHELONMENT.  Echelonment  or  subordination  of  enemy  elements  such 
as  battalions,  regiment,  divisions,  or  corps  to  specific  courses 
of  action. 

MAIN/ SUPPORTING  EFFORTS.  Where,  when,  and  in  what  strength  main 
and  supporting  efforts  will  occur  related  to  specific  enemy 
courses  of  action. 

FIRES  (including  air  support).  Where,  when,  and  in  what  strength 
fires  and  types  of  fires  will  occur  related  to  specific  enemy 
courses  of  action. 

TIME/DISTANCE  FACTORS.  Movement  in  terms  of  times  and  distances, 
i.e.,  how  long  it  will  take  for  a  unit  to  get  from  point  A  to  B, 
related  to  specific  enemy  courses  of  action. 

THREAT  ADVANCE.  How  rapidly  forces  can  or  will  move  related  to 
specific  enemy  courses  of  action,  and  influenced  by  doctrine  or 
terrain. 

PROBABILITY.  Probability  of  occurrence  for  enemy  courses  of 
action,  i.e.,  most  and  least  likely  COA  with  rationale,  in  the 
area  of  interest. 

Level  2:  ANALYSIS  OF  PROBABLE  ECOAs 

Probable  enemy  courses  of  action  are  divided  into  information 
items : 

Level  3 : 

ENEMY  STRENGTHS.  Specific  aspects  of  each  enemy  COA,  i.e.,  force 
ratios  or  terrain  which  are  to  their  advantage. 

ENEMY  VULNERABILITIES.  Specific  aspects  of  each  enemy  COA,  i.e., 
force  ratios  or  terrain  which  can  make  them  vulnerable. 

FRIENDLY  HIGH  VALUE  TARGETS.  Potential  friendly  high  value 
targets,  i.e.,  friendly  units  or  key  terrain  held  by  friendly 
units  that  may  influence  specific  enemy  courses  of  action. 

ENEMY  INTENTIONS.  Independent  of  the  assessed  probability, 
information  on  which  courses  of  action  the  enemy  intends  to 
execute . 

Level  is  ENEMY  ACTIVITIES  AFFECTING  OPERATIONS  SECURITY 

Specialized  enemy  activities  that  may  influence  command 
operations  security  posture  in  both  forward  and  rear  battlefield 
areas  are  delineated  into  five  topical  groups:  reconnaissance 
(RECCE)/intelligence,  radio  electronic  combat  (REC) ,  special 
operations,  friendly  vulnerabilities,  and  deception. 
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Level  2:  EN  RECCE/ INTELLIGENCE 


Enemy  reconnaissance  and  intelligence  activities  that  may 
influence  command  operations  security  posture  are  divided  into 
three  information  items: 

Level  3: 

EN  RECCE/ INTELLIGENCE  CAPABILITIES ,  i.e.,  imagery  or  human 
intelligence  that  may  influence  how  the  friendly  command  conducts 
operations. 

RECENT  RECCE/ INTELLIGENCE  ACTIVITIES/INDICATORS.  Most  current 
and  significant  enemy  intelligence  activities  and  indicators  that 
may  influence  command  operations  security  posture. 

EFFECTS  OF  EN  INTELLIGENCE  ON  FR  OPERATIONS.  Effects,  as 
determined  either  by  friendly  analysis  or  enemy  information,  of 
enemy  RECCE  and  intelligence  activities  on  our  operations. 

Level  2:  ENEMY  RADIO  ELECTRONIC  COMBAT 

Enemy  jamming  and  signals  collection  activities  that  may 
influence  the  command  operations  security  posture  is  divided  into 
three  information  items: 

Level  3: 

REC  CAPABILITIES.  Enemy  radio  and  electronic  warfare 
capabilities  that  may  influence  how  the  command  conducts 
operations. 

RECENT  AND  SIGNIFICANT  REC  ACTIVITIES.  Most  current  and 
significant  enemy  radio  and  electronic  warfare  activities  that 
may  influence  how  the  command  conducts  operations. 

EFFECTS  OF  REC  ON  FR  OPERATIONS.  Effects,  determined  by  either 
friendly  analysis  or  enemy  information,  of  enemy  Radio 
Electronic  Combat  activities  on  friendly  operations. 

Level  2:  EN  SPECIAL  OPERATIONS 

Enemy  special  operations  activities,  i.e.,  special  operating 
forces  and  espionage/sabotage  operations,  that  may  influence 
command  operations  security  posture  is  divided  into  three 
information  items: 

Level  3: 

ENEMY  SPECIAL  OPERATIONS  CAPABILITIES.  Enemy's  special 
operations  capabilities,  i.e.,  special  operating  forces  and 
espionage/sabotage  resources,  that  may  influence  the  command 
operations  security  posture. 
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RECENT/SIGNIFICANT  EN  SPECIAL  OPERATIONS.  Enemy's  recent  special 
operations,  i.e.,  special  operating  forces,  espionage,  or 
sabotage . 

EFFECTS  OF  EN  SPECIAL  OPERATIONS  ON  FR  OPERATIONS.  Effects, 
determined  either  by  friendly  analysis  or  enemy  information,  of 
enemy  special  operations  on  friendly  operations. 

Level  2:  FRIENDLY  VULNERABILITIES 

Friendly  vulnerabilities  determined  either  by  the  enemy  or 
friendly  analysis,  that  may  influence  command  operations  security 
posture,  is  divided  into  two  information  items: 

Level  3: 

FRIENDLY  HIGH  VALUE  TARGETS.  Friendly  vulnerability  information, 
derived  from  either  friendly  analysis  or  enemy  information,  that 
concerns  our  own  potentially  high  value  targets. 

EFFECTS  OF  FRIENDLY  VULNERABILITIES  ON  FRIENDLY  OPERATIONS. 
Vulnerability  information,  derived  from  friendly  analysis  or 
enemy  information,  that  concerns  the  effects  of  our 
vulnerabilities  on  current  or  projected  operations. 

Level  2:  DECEPTION 

Enemy  deception  or  spoofing  activities,  that  may  influence 
command  operations  security  posture,  are  divided  into  three 
information  items: 

Level  3: 

DECEPTION  CAPABILITIES.  Present  enemy  deception  capabilities 
that  may  influence  command  operations  security  posture.  These 
may  contain  numerous  specific  data  possibilities  that  need  not 
be  further  specified. 

RECENT/SIGNIFICANT  DECEPTION  ACTIVITIES.  Current  and  significant 
enemy  deception  or  spoofing  activities  that  may  influence  command 
operations  security  posture  or  friendly  combat  operations. 

EFFECTS  OF  DECEPTION  ON  FRIENDLY  OPERATIONS.  Effects,  determined 
by  friendly  analysis  or  enemy  information,  of  enemy  deception  of 
spoofing  activities  on  our  operations. 
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APPENDIX  A-4 


TIMELINESS  DIMENSION  SCALE 

TIMELINESS  -  is  a  measure  of  whether  the  information  item  was 
received  in  time  for  the  user  to  take  action  on  it.  Timeliness 
applies  to  situations  where  there  is  no  stated  operational 
deadline  for  the  information,  but  a  window  exists  during  which 
the  user  must  take  action  based  on  that  specific  information. 

Timeliness  Scale: 

1.  Item  received  and  user  had  ample  time  to  take  operational 
action  within  the  time  window. 

2.  Item  received  in  time  to  take  action,  but  user  had  to  rush. 

3.  Item  received  in  time  to  take  action,  but  user  had  to  rush 
and  use  additional  resources. 

4.  Item  received  too  late  for  user  to  take  action. 

5.  Item  not  received. 
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APPENDIX  A-5 


FREQUENCY  DIMENSION  SCALE 

FREQUENCY  -  is  a  measure  of  how  often  the  information  item  needs 
to  be  provided  to  a  user  in  order  for  the  user  to  carry  out 
designated  routine  functions.  FREQUENCY  involves  routine 
information  delivered  according  to  an  SOP,  by  a  specified  time  or 
in  a  specified  cycle. 

Frequency  Scale; 

1.  Item  received  often  enough. 

2.  Item  received  more  than  necessary,  but  did  not  disrupt  SOP 
functions. 

3.  Item  received  more  than  necessary  and  disrupted  SOP 
functions. 

4.  Item  not  received  often  enough. 

5.  Item  not  received. 
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APPENDIX  A-6 


OPERATIONAL  PERSPECTIVE  DIMENSION  SCALE 

OPERATIONAL  PERSPECTIVE  -  is  the  measure  of  how  well  the 
information  item  is  put  in  the  context  of  current  or  future 
friendly  force  operations.  To  contribute  to  OPERATIONAL 
PERSPECTIVE,  the  following  context  characterizations  apply: 

Area  of  operations 
Area  of  interest 

Time/Phasing/Duration  of  the  operation 
Missions  of  higher,  lower  or  adjacent  units 
Unit  contingency  plans  in  place  or  in  development 
Friendly  capabilities  related  to  current  or  future  ops 
Enemy  capabilities  related  to  current  or  future  ops 

Operational  Perspective  Scale: 

1.  Operational  perspective  identified  a  new  or  different 
perspective  on  current  or  future  operations. 

2.  Operational  perspective  related  to  key  aspects  of  current  or 
future  operations. 

3.  Operational  perspective  related  to  less  significant  aspects 
of  current  or  future  operations  or  were  limited  in  scope. 

4.  Operational  perspective  did  not  relate  to  current  or  future 
operations. 

5.  Item  not  placed  in  any  operational  context. 
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APPENDIX  A-7 


CLARITY  DIMENSION  SCALE 

CLARITY  -  is  a  measure  of  how  easily  the  content  of  the 
information  could  be  grasped,  understood,  comprehended  or 
followed  by  the  user,  without  regard  to  the  presentation  format 
The  following  factors  detract  from  the  CLARITY  of  information 
items : 

Poorly  organized 
Too  much  jargon 
Too  detailed 
Too  general 
Emphasis  in  wrong 
Too  long 

Too  many  acronyms 
Clarity  Scale: 

1.  Item  easily  understood  without  effort  or  clarification. 

2.  Item  easily  understood  with  comparison  to  other 
information  readily  available  to  user. 

3.  Item  understandable  with  time,  effort,  and/or  clarification 

4.  Item  took  extensive  time,  effort,  and/or  clarification  to 
understand . 

5.  Item  not  understandable. 


Too  technical 

Content  level  inappropriate 
for  recipient 

Inappropriate  presentation 
areas  media 

Too  abbreviated 
Poor  use  of  graphics 


A-23 


APPENDIX  A- 8 


COMPLETENESS  DIMENSION  SCALE 

COMPLETENESS  -  is  a  measure  of  the  factual  content  of  the 
information  item  as  represented  by  the  5  W's,  SALUTE  and/or 
METT-T.  The  absence  of  facts  does  not  distract  from 
completeness  if  the  gaps  are  identified.  The  following  relate  to 
the  COMPLETENESS  of  information: 


METT-T 

SALUTE 

5  W's 

Mission 

Size 

Who 

Enemy 

Activity 

What 

Terrain  (Weather) 

Location 

Where 

Troops 

Unit 

When 

Time  Available 

Time 

Why 

Equipment 

How 

Completeness  Scale: 

1.  No  factual  omissions/gaps. 

2.  Fully-identified  and  explained  factual  omissions/gaps. 

3.  Partially-identified  and  explained  factual  omissions/gaps. 

4.  Unexplained  factual  omissions/gaps. 

5.  Information  unusable  because  there  were  too  may  unexplained 
omissions/gaps . 
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APPENDIX  A-9 


INSTRUCTIONS  FOR  SETTING  BENCHMARKS 

Determining  the  Benchmarks; 

1.  For  any  dimension,  review  the  five  levels  of  performance 
indicated  by  the  scales. 

2.  Select  the  point  on  the  scale  that  best  describes  the  minimum 
level  of  performance  which  would  be  acceptable  to  you. 

3.  Draw  a  line  under  that  scale  number. 

4.  Review  the  items  under  that  line  to  ensure  they  describe 
performance  unacceptable  to  you.  If  not,  redetermine  the 
benchmark. 
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APPENDIX  B 


SUPPLEMENTAL  MATERIALS  FOR  RATING 
MILITARY  INTELLIGENCE  PERFORMANCE 

B-l  Performance  Rating  Form,  Part  1  and  2 

B-2  Instructions  for  Rating  Military  Intelligence  Performance 


Performance  Rating  Form 

(part  1  of  2) 


Performance  Rating  Form 

(part  2  of  2) 


APPENDIX  B-2 


INSTRUCTIONS  FOR  RATING 
MILITARY  INTELLIGENCE  PERFORMANCE 


A  copy  of  the  Performance  Rating  Form,  part  1  and  part  2,  and 
scales  and  definitions  for  the  five  dimensions:  timeliness, 
frequency,  operational  perspective,  clarity,  and  completeness  are 
required  for  rating. 


Step  1. 


Step  2. 


Step  3. 


Step  4. 


Step  5. 


Step  6. 


Step  7. 


Review  the  scales  and  the  definitions  of  the  scales 
(Appendix  A-4  thru  A-8) .  The  definitions  should  be 
consulted  during  the  rating,  as  necessary. 

Rate  all  the  information  items  for  timeliness.  For 
each  item,  place  the  number  of  the  scale  level 
selected  in  the  "timeliness  rating"  column  in  the 
space  next  to  the  information  item.  For  all 
information  items  rated  5,  not  received,  draw  a  line 
through  rest  of  the  spaces  in  the  row,  for  those 
items.  Those  information  items  cannot  be  rated  on 
the  other  dimensions. 

Rate  all  information  items  for  frequency.  Place  the 
number  of  the  scale  level  selected  in  the  "frequency 
rating"  column  in  the  row  for  the  information  item. 

Rate  all  information  items  for  operational 
perspective.  Use  the  same  procedures  as  previously, 
placing  the  rating  in  the  appropriate  row  and  column. 
Where  there  was  a  deficiency  refer  to  the  sample 
deficiencies.  If  the  deficiency  is  described,  place 
the  letter  for  that  deficiency  in  the  "operational 
perspective — deficiencies"  column  in  the  row  for  the 
deficient  item.  An  item  may  have  more  than  one 
deficiency.  If  the  deficiency  is  not  in  the  sample 
list,  write  in  the  deficiency  you  observed. 

Rate  all  the  information  items  for  clarity  using  the 
same  procedures  as  for  the  other  dimensions.  Place 
your  rating  in  the  appropriate  row  and  column  and 
identify  the  deficiencies  where  necessary. 

Rate  all  the  information  items  for  completeness  using 
the  same  procedures  as  for  the  other  dimensions. 

Place  your  rating  in  the  appropriate  row  and  column 
and  identify  the  deficiencies  where  necessary. 

Review  the  rating  form  to  ensure  that  all  items  have 
been  rated  for  each  dimension. 

Reestablish  the  benchmark  for  each  dimension. 
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Step  8. 


Step  9. 


Turn  in  the  ratings  according  to  the  previously 
determined  procedure. 
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APPENDIX  C 

SUPPLEMENTAL  MATERIALS  FOR  DIAGNOSING 
MILITARY  INTELLIGENCE  PERFORMANCE 

C-l  Deficiency  Consolidation  Worksheets 

C-2  Instructions  for  the  Use  of  the  Deficiency  Consolidation 
Worksheets 

C-3  Planned  Inquiry  Guide  Form 

C-4  Instructions  for  the  Use  of  the  Planned  Inquiry  Guide 
C-5  Item  Deficiency  to  Source  Matrix 

C-6  Instructions  for  the  Use  of  the  Diagnostic  Question 
Inventory  and  Flow  Charts 

C-7  Diagnostic  Question  Inventory  and  Flow  Chart:  Timeliness 

C-8  Diagnostic  Question  Inventory  and  Flow  Chart:  Frequency 

C-9  Diagnostic  Question  Inventory  and  Flow  Chart:  Operational 

Perspective 

0-10  Diagnostic  Question  Inventory  and  Flow  Chart:  Clarity 

C-ll  Diagnostic  Question  Inventory  and  Flow  Chart:  Completeness 

C-12  Diagnostic  Record 

C-13  Instructions  for  the  Use  of  the  Diagnostic  Record  Form 
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APPENDIX  C-l 


DEFICIENCY  CONSOLIDATION  WORKSHEETS 
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DEFICIENCY  CONSOLIDATION  WORKSHEET 


UDAT10N  WORKSHEET 


APPENDIX  C-2 


INSTRUCTIONS  FOR  THE  USE  OF  THE 
DEFICIENCY  CONSOLIDATION  WORKSHEET 

There  are  four  different  worksheets,  one  for  each  of  the  four 
major  information  categories,  that  must  be  used.  This  worksheet 
helps  to  consolidate  data  from  the  observer-participant 
Performance  Ratings  Forms.  A  deficient  item  is  one  that  falls 
below  the  user's  benchmark.  The  following  steps  are  suggested 
when  there  are  so  many  Performance  Rating  Forms  to  be 
consolidated  that  a  mechanical  procedure  is  required. 

Step  1.  On  each  Performance  Rating  Form,  circle  the  deficient 

ratings  for  each  of  the  rated  dimensions.  Complete 
this  step  before  consolidating  the  data. 

Step  2.  Record  in  the  appropriate  cell  of  the  worksheet  the 

user  who  rated  the  item  deficient.  Do  this  for  each 
of  the  dimensions.  For  the  timeliness  dimension 
deficiencies,  identify  when  the  information  item  was 
not  received. 

Step  3.  Complete  step  2  for  all  observer-participant 

worksheets.  Make  some  kind  of  mark  on  the  worksheet 
as  you  finish  recording  the  data  to  ensure  the  same 
information  is  not  recorded  more  than  once. 
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APPENDIX  C-3 

PLANNED  INQUIRY  GUIDE  FORM 
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APPENDIX  C-4 


INSTRUCTIONS  FOR  THE  USE  OF  THE 
PLANNED  INQUIRY  GUIDE 


The  Planned  Inquiry  Guide  is  used  in  conjunction  with  the 
Deficiency  Consolidation  Worksheet,  the  Information  Item  to 
Source  Matrix,  and  the  Diagnostic  Question  Inventory  and  Flow 
Charts.  Its  purpose  is  to  systematize  and  document  the 
diagnostic  strategy. 


Step  1. 


Step  2. 


Step  3. 


Step  4. 


Step  5. 


Step  6. 


Determine  which  deficiencies  will  be  diagnosed  using 
the  Deficiency  Consolidation  Worksheet.  Record  the 
first  deficiency  on  the  worksheet  in  the 
"deficiencies  or  areas  to  diagnose"  column. 

If,  based  on  experience,  you  have  any  ideas  on  what 
the  possible  cause  of  the  deficiency  may  be,  record 
them.  If  not,  leave  the  "potential  cause"  column 
blank. 

Use  the  Information  Item  Deficiency  to  Source  Matrix 
to  determine  the  best  place  to  begin  gathering  data. 
Record  the  location  in  the  "where  to  start"  column. 

Review  the  Diagnostic  Question  Inventory  and  flow 
charts  to  select  what  kind  of  questions  needed  to  be 
answered  to  identify  the  cause  of  the  deficiency. 
Record  those  questions  and  any  others  you  may  want 
answered  in  the  "questions  to  answer"  column. 

Repeat  the  above  steps  for  each  deficiency  you  intend 
to  diagnose. 

When  you  have  finished  the  planned  inquiry,  begin 
collection  data. 
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APPENDIX  C-5 


ITEM  DEFICIENCY  TO  SOURCE  MATRIX 
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INFORMATION  ITEM  DEFICIENCY  TO  SOURCE  MATRIX 


Information  Item 

Category 

Items  Omitted 

Poor  Quality  Items 

Operational 

Perspective 

Clarity  Completeness 

BATTELFIELD  AREA 

G2  Ops 

G2 

ASPS  G2  Ops 

Weather 

USAF  WO 

G2 

G2  Ops  G2  Ops 

Terrain 

ENG  TM 

G2 

ASPS 

Area  conditions 

ASPS 

G2 

ASPS 

ENSIT 

G2  Ops 

G2 

G2  Ops 

Disposition  &  Comp. 

G2  Ops/CM&D 

G2 

G2  Ops 

Strength 

G2  Ops/CM&D 

G2 

G2  Ops 

Activities 

G2  Ops/CM&D 

G2 

G2  Ops 

ENCOAS 

ASPS 

G2 

ASPS 

Possible  COAs 

ASPS 

G2 

ASPS 

Other  Info  on  Prob.  COAs 

ASPS/CM&D 

G2 

ASPS 

OPSEC 

G2/CI  TEAM 

G2 

G2  Ops 

EN  Recce/Intel 

ASPS/CM&D 

G2 

ASPS/G2  Ops 

EN  REC 

ASPS/CM&D 

G2 

ASPS/G2  Ops 

EN  Special  Ops 

ASPS/CM&D 

G2 

ASPS/G2  Ops 

FR  Vulnerabilities 

G2/ASPS 

G2 

ASPS/G2  Ops 

Deception 

G2/ASPS 

G2 

ASPS/G2  Ops 

UNIT  SPECIFIC  COMMENTS: 
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APPENDIX  C-6 


INSTRUCTIONS  FOR  THE  USE  OF  THE 
DIAGNOSTIC  QUESTION  INVENTORY  AND  FLOW  CHARTS 

Both  the  diagnostic  question  inventory  and  the  flow  charts 
present  a  logical  way  to  identify  the  questions  needed  to 
determine  the  cause (s)  of  deficiencies  in  the  intelligence 
information  production  system.  The  flow  charts  provide  a  rapid 
view  of  possible  general  questions  that  might  be  used,  and  the 
relationships  to  other  questions  which  may  have  to  be  answered  to 
obtain  a  complete  diagnosis.  The  Diagnostic  Question  Inventory 
presents  a  more  detailed  breakdown  of  the  general  questions.  The 
general  questions,  along  with  their  breakdown  and  the  specific 
information  item,  can  provide  some  very  focused  questions  for 
diagnostics. 

Diagnostic  Question  Inventory 

The  person  doing  the  diagnosis  of  military  intelligence 
information  production  must  be  experienced  MI.  That 
background,  the  diagnostic  procedures,  and  the  questions  in 
this  inventory  will  help  determine  why  observer-participants 
rated  some  of  the  information  they  received  as  unacceptable. 

To  determine  the  cause (s)  of  the  deficiencies,  the 
diagnostician  must  determine  the  answers  to  a  series  of 
questions.  The  inventory  provides  the  diagnostician  an  aid  or 
guide  in  determining  what  questions,  in  what  sequence,  he  must 
answer  in  conducting  his  inquiry.  The  inventory  is  used  to 
assist  the  diagnostician  in  filling  out  column  four  "questions 
which  must  be  answered"  of  the  Planned  Inquiry  Guide. 

Organization  of  the  Diagnostic  Question  Inventory 

The  inventory  is  in  five  sections,  each  corresponding  to  one  of 
the  dimensions  for  which  an  information  item  could  be 
deficient:  timeliness,  frequency,  operational  perspective, 
clarity,  and  completeness.  There  are  a  set  of  primary 
questions  with  directions  on  where  to  find  the  next  set  of 
questions.  The  directions  are  based  on  the  answer  to  the 
primary  question. 

There  are  sets  of  secondary  questions  which  should  be  answered 
in  order  to  identify  the  cause  of  the  deficiency. 

Since  the  two  deficiency  areas,  frequency  and  completeness  have 
questions  in  common,  the  questions  are  not  repeated  for  each 
deficiency  area.  For  example,  a  primary  question  for 
diagnosing  completeness  may  direct  you  to  go  to  question  3  of 
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the  frequency  deficiency  diagnosis.  That  is  because,  with 
proper  modification,  the  same  questions  are  relevant  to 
deficiencies  in  both  areas. 

Flow  Charts  for  the  Diagnostic  Question  Inventory 

For  each  deficiency  area  there  is  a  flow  chart  of  the 
questions  in  the  inventory.  The  flow  charts  provide  a  view  of 
the  entire  diagnostic  questioning  for  each  deficiency 
dimension.  The  numbers  on  the  flow  charts  refer  to  the 
questions  in  the  inventory.  For  example,  the  number  5b  in  the 
timeliness  deficiency  flow  chart  refers  to  the  b  secondary 
questions  in  primary  question  5  for  the  timeliness  deficiency 
dimension.  In  the  case  of  completeness  which  has  overlapping 
questions  with  timeliness,  a  number  with  two  letters  following 
is  direction  to  go  to  a  different  flow  chart.  For  example,  2af 
in  the  completeness  flow  chart  means  to  go  to  the  secondary 
questions  a  for  question  2  on  the  frequency  flow  chart. 

On  the  flow  charts  for  operational  perspective,  clarity,  and 
completeness  are  the  specific  deficiencies  which  may  have 
occurred.  They  are  listed  only  as  reminders. 

The  flow  charts  can  be  used  to  gain  a  fast  overview  of  the 
questions  for  a  deficiency  dimension  and  skip  directly  into  the 
middle  of  the  inventory  instead  of  reading  through  each  set  of 
questions. 

FINAL  NOTE:  THE  DIAGNOSTIC  QUESTION  INVENTORY  AND  FLOW  CHARTS 
ARE  GUIDES.  THEY  ARE  NOT  INTENDED  TO  BE  A  COMPREHENSIVE  LIST  OF 
ALL  THE  QUESTIONS  WHICH  COULD  POSSIBLY  BE  ASKED  DURING  THE 
DIAGNOSTIC  PROCESS. 
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APPENDIX  C-7 


DIAGNOSTIC  QUESTION  INVENTORY  AND  FLOW  CHART: 

TIMELINESS 

Timeliness  is  unusual  in  that  if  you  find  a  possible  cause  for 
deficiency,  correcting  it  might  not  provide  enough  additional 
time  to  have  prevented  the  deficiency.  You  may  be  dealing  with 
multiple  causes,  which  together  contributed  to  the  deficiency. 

As  a  result,  you  will  not  always  stop  asking  questions  when  you 
have  discovered  a  cause.  Follow  questioning  in  sequence  unless 
you  are  to  go  to  a  different  question. 

The  first  set  of  questions  is  to  determine  whether  something  went 
wrong  with  dissemination.  In  other  words,  something  got  fouled 
up  getting  the  information  out  to  the  user.  Stop  asking 
questions  about  an  information  item  when  you  have  fully 
identified  what  went  wrong.  Go  through  each  information  item  on 
your  data  sheet.  It  may  become  apparent  that  the  same  problem 
is  responsible  for  a  deficiency  in  many  items.  Use  common  sense 
on  when  to  stop  asking  questions.  Item  la  below  is  a  good 
example.  If  there  are  no  procedures  for  prioritizing 
information,  that  may  affect  all  deficient  items,  that  doesn't 
mean  questions  Id  and  le  should  not  be  asked  since  they  have 
nothing  to  do  with  prioritization. 

Questions  1  and  2  are  concerned  with  the  dissemination  of 
information  from  the  MI  producer  to  the  user  without  respect  to 
form;  it  makes  no  difference  if  dissemination  is  a  formal 
briefing,  phone  call,  or  radio  transmission. 

1.  Was  information  item  sent  to  the  user?  If  no,  go  to  Question 

3.  If  yes,  then: 

a.  Was  a  priority  placed  on  getting  the  information  sent? 

If  no,  check  to  see  if  there  were  adequate  procedures  for 
prioritizing  communications.  If  there  weren’t,  maybe 
there  should  be.  If  there  were,  why  weren't  they 
followed? 

b.  Did  the  priority  cause  other  information  to  be  sent 
first,  thus  delaying  dissemination?  If  yes,  was 
information  prioritized  correctly,  were  procedures 
followed,  or  was  everything  done  correctly,  but  more 
important  information  needed  to  be  sent  out  first? 

c.  Did  the  person  responsible  for  dissemination  respond  to 
the  priority?  If  no,  why  not?  If  yes,  the  problem  may 
be  identified  by  Questions  id  or  le. 
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d.  Did  the  person  responsible  for  dissemination  select  the 
best  means/alternate  means  to  send  the  information?  If 
no,  look  again  at  procedures  to  determine  if  they  exist, 
cover  the  situation,  and  were  followed. 

e.  Was  it  confirmed  that  the  user  received  the  information? 
If  no,  did  procedures  exist,  did  they  cover  the 
situation,  and  were  they  followed?  Could  confirmation  be 
carried  out? 

2.  If  you  identified  a  possible  cause  for  timeliness  deficiency, 
would  correcting  it  have  resulted  in  enough  time  being  saved 
to  have  prevented  the  deficiency?  If  yes,  have  you 
identified  the  entire  cause?  If  you  didn’t  identify  a  cause, 
or  if  correcting  the  cause  you  identified  wouldn't  have  saved 
enough  time,  you  will  have  to  begin  asking  about  information 
processing.  Go  to  Question  5. 

3.  The  information  item  was  not  sent  to  the  user.  Did  the  item 
get  sent  to  the  person  responsible  for  disseminating  it?  If 
yes,  why  didn't  it  get  sent?  If  no,  the  cause  of  the 
deficiency  is  almost  certainly  not  in  the  dissemination 
process. 

4.  Did  the  organization  get  the  information  required  to  produce 
the  information  item?  If  no,  determine  the  organization  that 
should  have  sent  the  information.  You  must  go  to  that 
organization  and  ask  questions  on  timeliness. 

5.  Did  data  received  meet  organization  standards  necessary  to 
effectively  process  it?  No  matter  what  the  answer,  proceed 
with  questioning.  However,  if  no,  identify  and  examine  the 
organization (s)  from  which  the  data  came. 

a.  Was  the  potential  significance  of  the  data  or  information 
recognized?  If  no,  it  may  be  that  the  importance  of  the 
data  could  be  recognized  only  after  it  had  been  processed 
or  placed  in  context  with  other  information.  If  that  is 
the  case,  go  to  Question  5d.  If  the  potential 
significance  could  have  been  recognized,  then  why  wasn't 
it?  Were  procedures  established  to  screen  incoming  data 
for  significance?  Did  the  person  have  the  experience 
necessary  to  do  the  screening?  Did  he  have  the 
training?  Where  job  aids  available  to  assist  in  the 
screening?  Were  aids  possible?  If  available,  were  they 
used? 

b.  Was  the  significance  passed  on  to  the  people  required  to 
process  the  information?  If  no,  why  not? 
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c.  How  long  did  it  take  to  process  the  information?  The 
organization  may  have  done  everything  possible,  yet  atiJl 
was  unable  to  get  information  out  in  time  to  be  of  use. 

If  answers  to  the  next  four  questions  are  negative, 
identify  the  organization  that  sent  the  information  to 
determine  if  the  problem  can  be  solved  there.  These 
questions  can  be  asked  in  any  order,  but  all  need  to  be 
considered. 

(1)  Would  additional  resources  speed  information 
processing?  What  resources?  More  people?  How 
many?  More  equipment?  What  kind?  Automated? 
Manual?  Job  aids?  What  kind?  Would  additional 
resources  make  a  difference? 

(2)  Would  more  effective  resources  speed  information 
processing?  Do  all  personnel  have  necessary 
training  and  experience?  Is  correct  equipment 
available  and  working?  Would  more  effective 
resources  make  a  difference? 

(3)  Would  more  effective  use  of  resources  speed 
information  processing?  Are  proper  procedures 
followed?  Could  procedures  be  improved?  What 
procedures?  Is  unnecessary  work  done?  Are  all 
personnel  used  properly?  Equipment?  Would  more 
effective  use  of  resources  make  a  difference? 

(4)  Would  changes  in  any  combination  of  the  three  above 
make  a  difference?  If  changes  would  have  entirely 
prevented  the  timeliness  deficiency,  has  the 
complete  cause  been  identified? 

d.  Was  the  significance  of  the  processed  information 
recognized?  If  no,  were  procedures  established  to 
identify  significant  information?  Were  they  followed? 
Were  there  job  aids  to  assist?  Were  they  used?  Were 
personnel  properly  trained?  Did  they  have  adequate 
experience? 

e.  Was  the  information  item  prioritized  for  dissemination? 
Were  prioritization  procedures  established  and  used? 

f.  Was  the  information  sent  to  the  dissemination  point?  If 
not ,  why? 

Even  when  cause (s)  are  identified,  correction  must  make  a 
difference.  Would  the  time  saved,  including  time  saved  by 
correcting  dissemination  deficiencies,  have  eliminated  the 
timeliness  deficiency?  If  not,  continue  questioning  the 
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organization  which  sent  the  information.  If  all  questions  are 
answered  "yes,"  the  problem  is  not  getting  information  to  the 
organization  fast  enough.  Do  a  timeliness  diagnosis  on  the 
organization  from  which  the  information  is  coming. 
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APPENDIX  08 


DIAGNOSTIC  QUESTION  INVENTORY  AND  FLOW  CHART: 

FREQUENCY 

There  are  three  ways  an  information  item  can  be  rated 
unacceptable,  based  on  a  frequency  deficiency.  First, 
information  was  not  received.  Second,  it  was  received,  but  the 
user  did  not  get  it  as  frequently  as  needed;  implicit  in  this 
deficiency  is  that  information  may  be  late.  Finally,  information 
could  come  in  more  frequently  than  required,  disrupting 
operations . 

The  first  set  of  questions  is  to  determine  whether  something  went 
wrong  with  the  dissemination,  without  respect  to  form;  it  doesn't 
matter  if  dissemination  is  a  formal  brief,  phone  call,  or  radio 
transmission. 

1.  Did  the  user  receive  the  informations  If  yes,  go  to 
Questions  4  and  5. 

a.  If  no,  was  item  sent  to  the  person  responsible  for 
disseminating  it?  If  not,  diagnose  the  organization  that 
should  have  sent  it. 

b.  If  the  item  was  received,  why  wasn't  it  forwarded  to  the 
user?  Did  anyone  know  the  information  was  there  to  be 
sent  on?  Was  a  conscious  decision  made  not  to  send  it 
on?  If  so,  why? 

c.  If  it  was  forwarded,  did  it  go  to  the  correct  user?  Was 
it  confirmed  that  the  user  had  received  the  information? 
Were  there  procedures  on  confirmation?  Were  they  used? 
Was  confirmation  possible? 

2.  Did  the  organization  responsible  for  producing  the 
information  item  have  information  required  for  processing 
into  the  item?  If  yes,  go  to  2c. 

a.  If  not,  did  anyone  realize  they  didn't  have  the 
information?  Were  any  quality  control  procedures  in 
effect  to  identify  when  or  what  information  might  be 
missing?  Were  quality  control  procedures  used?  Why 
didn't  they  work?  Did  anyone  know  the  information  was 
needed  to  meet  a  suspense? 

b.  Were  requests  made  for  the  information?  If  not,  why?  If 
requests  were  made,  were  they  made  according  to 
procedure?  Were  requests  followed-up?  Confirmed?  Was 
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the  suspense  passed  on?  If  information  was  requested  and 
not  received,  diagnose  the  organization  responsible  for 
collecting  the  information.  Go  to  Question  3. 

c.  If  they  had  the  information,  did  it  meet  standards 
required  to  adequately  process?  If  no,  did  it  have  any 
effect  on  not  meeting  the  suspense?  If  yes,  what  was  the 
impact?  (This  question  is  to  help  determine  where  in  the 
organization  you  might  want  to  go,  or  what  other 
questions  you  may  be  able  to  eliminate  to  identify  the 
cause  of  the  deficiency) . 

d.  If  the  information  did  not  get  processed,  was  the  crew 
aware  of  the  suspense?  Was  there  time  enough  to  produce 
the  information  item?  Were  there  higher  priority  items 
to  be  taken  care  of?  Were  there  enough  resources? 

Human?  Material?  Were  there  too  many  resources?  Were 
there  procedures  in  place  to  ensure  work  got  done  and  the 
suspense  met?  Did  the  procedures  work? 

e.  If  the  information  was  processed,  were  results  reported? 
If  not,  why?  If  so,  were  they  removed  from  the 
dissemination?  Why? 

3.  We  know  from  item  2b  that  requests  for  the  information  were 
made;  was  the  data  collected?  If  not,  why?  If  so,  why 
wasn't  it  passed  on? 

a.  Were  data  requirements  passed  on  to  staff  responsible  for 
collecting?  Were  the  requirements  correctly  translated 
into  collection  requirements?  Were  there  higher 
collection  priorities  which  precluded  the  collection  of 
the  data?  Were  assets  available  for  collecting  the  data? 
Were  all  possible  assets  considered?  Were  they 
assigned?  Was  an  attempt  made  to  collect  the  data? 

b.  If  collected,  was  collection  acknowledged?  Were  there 
procedures  for  passing  on  the  data?  Were  they  followed? 

4.  Was  the  item  received  too  frequently,  disrupting  the 
operations? 

a.  Is  there  guidance  concerning  the  frequency  at  which 
information  is  to  be  provided?  Guidance  can  be  in  the 
form  of  SOP,  supervisory  input,  or  user  feedback.  Was 
guidance  used? 

b.  Were  user's  frequency  requirements  known?  Was  action 
taken  to  meet  them?  Were  alternative  actions  possible? 

5.  Was  the  information  item  received  less  frequently  than  the 
user  required?  Implicit  in  this  situation  is  that 
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information  could  have  been  consistently  provided  late; 

specific  instances  would  be  evaluated  as  a  timeliness 

deficiency. 

a.  Was  there  guidance  concerning  the  frequency  the 
information  was  provided  the  user?  This  guidance  could 
be  in  the  form  of  SOP,  supervisory  input,  or  feedback 
from  the  user.  Was  the  guidance  used? 

b.  Did  the  data  to  be  processed  into  the  information  item 
meet  the  standards  required  for  processing?  If  not, 
would  it  have  contributed  to  the  information  being 
provided  less  frequently  than  required?  If  so,  how? 

c.  Were  there  higher  priority  information  requirements  that 
needed  to  be  processed?  Were  alternative  resources 
available  to  produce  the  information?  Were  they 
considered?  Did  the  processing  procedure  contribute  to 
delay  in  processing? 

d.  Was  the  deficiency  in  meeting  the  frequency  standards  as 
a  result  of  the  dissemination  procedure?  Questions  la 
through  le  from  the  timeliness  diagnosis  can  be  used  for 
this  diagnosis. 

e.  Were  the  user's  frequency  requirements  known?  Was 
action  taken  to  meet  them?  Were  alternative  actions 
possible? 
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APPENDIX  C-9 


DIAGNOSTIC  QUESTION  INVENTORY  AND  FLOW  CHART: 

OPERATIONAL  PERSPECTIVE 

An  item  can  be  deficient  because  operational  perspective  was  not 
provided  or  was  provided  ineffectively.  Aspect (s)  or  operational 
perspectives  not  provided  or  provided  poorly  should  be 
identified,  i.e.: 

Area  of  operations 

Time/phasing/duration  of  the  operation 

Missions  of  higher,  lower,  or  adjacent  units 

Unit  contingency  plans,  in  place  or  in  development 

Friendly  capabilities  related  to  current  or  future  operations 

Enemy  capabilities  related  to  current  or  future  operations 

1.  Was  the  information  item  placed  in  operational  perspective? 

If  no,  then: 

a.  Was  there  any  guidance  to  indicate  the  information  should 
have  been  put  into  the  specific  operational  context? 
Guidance  could  include  SOP,  doctrine,  suggestions  from 
supervisors  or  cohorts,  or  feedback  from  the  user. 
Training  and  experience  in  providing  operational  context 
should  be  considered.  Guidance  can  be  implicit  or 
explicit.  If  no,  why?  Go  to  Question  2  to  determine  if 
having  guidance  would  have  made  a  difference,  and  to 
Questions  3  and  4  to  determine  if  there  was  a  reporting 
or  dissemination  problem. 

b.  Was  there  either  explicit  or  implicit  guidance  that 
information  should  NOT  be  put  in  operational  perspective? 
The  purpose  of  this  question  is  to  determine  if  there  is 
a  disconnect  between  what  should  be  done  and  what  is 
done.  If  the  answer  is  yes,  note  the  problem  and  proceed 
to  Question  2.  The  problem  of  the  disconnect  will  be 
addressed  outside  the  diagnostics. 

2.  Were  the  necessary  factors  in  place  so  the  information  could 

adequately  be  put  in  operational  perspective? 

a.  Is  there  time  to  place  the  information  in  operational 
perspective?  Is  the  workload  such  that  adding  this 
attribute  is  traded-off  in  order  to  accomplish  other 
requirements? 

b.  Does  the  staff  have  the  training  or  experience  necessary 
to  put  information  into  operational  perspective? 

Consider  formal  military  training,  on-the-job  training, 
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and  experience  based  on  previous  jobs,  in  relation  to 
operational  perspective  in  general  and  the  deficient 
aspect  of  operational  perspective  in  specific. 

c.  Is  the  staff  knowledgeable  of  procedures  that  facilitate 
placing  information  in  operational  perspective?  Do  they 
use  them?  Procedures  might  include  wargaming,  consulting 
with  staff  external  to  the  group,  internal  discussion, 
and  brainstorming. 

d.  Does  the  staff  have  the  job  aids,  data  bases,  references, 
etc.,  which  would  help  them  place  information  in 
operational  perspective? 

If  information  was  in  operational  perspective  which  wasn't 
forwarded  to  the  user,  was  it  removed  from  the  report 
(without  regard  to  how  it  was  presented)?  If  yes,  why? 

If  operational  perspective  was  in  the  report,  why  was  it 
removed  from  the  dissemination? 
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APPENDIX  C-10 


DIAGNOSTIC  QUESTION  INVENTORY  AND  FLOW  CHART: 

CLARITY 

A  clarity  deficiency  concerns  information  which  is  difficult  .to 
understand  or  comprehend  by  the  user,  regardless  of  how 
information  is  presented.  Reason (s)  for  a  clarity  deficiency  may 
include  any  combination  of: 

Poor  organization 

Too  technical  or  detailed 

Too  long  or  redundant 

Too  general  or  abbreviated 

Too  much  jargon,  too  many  acronyms 

Wrong  areas  emphasized 

Content  level  inappropriate  for,  the  user 
Wrong  means  of  presentation 

Four  major  sets  of  questions  should  be  addressed: 

1.  Were  procedures/ guidance  available  on  how  to  prepare  and 
present  information?  This  could  include  message  formats, 
briefing  guides.  Army  writing  style  manuals,  field  manuals, 
and  internal  SOPs  within  the  unit  and  the  command.  Were  they 
used?  Did  they  contribute  to  the  problem?  Why  weren't  they 
used? 

2.  Was  the  audience  for  the  information  identified?  Was  it 
considered  when  producing  or  presenting  the  information?  If 
a  multiple  audience,  were  key  players  identified?  Was 
production  or  presentation  adjusted  for  each?  If  one  key 
player,  was  he  identified?  Were  his  unique  requirements 
identified  and  considered? 

3 .  Was  there,  quality  control  of  the  production?  Was  someone 
responsible  for  checking  the  clarity  of  the  information?  Was 
it  done?  Was  there  enough  time  for  quality  control?  If  it 
was  done,  were  corrective  actions  initiated?  Followed  up? 

4.  Was  the  staff  adequately  trained  to  produce  or  present  the 
information?  Did  they  understand  what  was  being 
produced/presented?  Was  it  SOP  to  use  jargon,  acronyms,  or 
technical  language?  Was  it  SOP  to  be  brief  or  detailed  in 
carrying  out  their  responsibilities?  Was  lack  of 
organization  a  common  characterization  or  a  unique  event? 
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APPENDIX  C-ll 


DIAGNOSTIC  QUESTION  INVENTORY  AND  FLOW  CHART: 

COMPLETENESS 

An  item  is  deficient  if  missing  factual  content  (5W/H,  SALUTE, 
METT-T) .  An  absence  of  facts,  however,  does  not  distract  if  the 
absence  is  identified  and  explained.  Diagnosis  must  be  guided  by 
specific  mission  fact(s). 

1.  Were  the  missing  facts  available  for  dissemination?  If  no, 
go  to  Question  2. 

a.  Why  didn't  they  get  sent  to  the  user?  Did  anyone  know 
they  were  there  to  be  sent  on?  Was  a  conscious  decision 
made  not  to  send  it  on? 

b.  If  they  were  forwarded,  did  they  go  to  the  correct  user? 
Was  it  confirmed  that  the  user  received  them?  Were  there 
procedures  for  confirmation?  Were  they  used?  Was 
confirmation  possible? 

2.  If  data  had  to  be  processed  in  some  way  to  develop  the 
factual  content  of  the  information  item,  did  the  staff  doing 
the  processing  have  the  necessary  data?  If  no,  diagnose  data 
collector  as  well  as  data  processor. 

a.  If  the  processor  didn't  have  the  data,  then  Questions  2a 
and  2b  in  the  frequency  analysis  are  appropriate. 

b.  If  the  processor  had  the  data,  why  wasn't  it  processed? 
Was  it  lost  in  the  normal  course  of  events?  Did  the 
staff  know  the  data  was  supposed  to  be  processed  for  the 
item?  Were  guidelines  or  references  available  to  help 
the  staff?  Did  they  have  necessary  training  or 
experience  to  carry  out  the  processing?  Was  someone 
responsible  for  quality  controlling  the  outputs?  Did  it 
get  done?  Were  corrections  for  deficiencies  noted  during 
quality  control  directed?  Was  the  direction  followed  up? 

c.  If  the  data  was  processed,  was  it  included  in  the 
reporting?  If  not,  why  not?  If  it  got  reported,  did  it 
get  lost  in  the  dissemination? 

3.  Was  the  data  collected?  With  minor  modifications,  Question  3 
of  the  frequency  diagnosis  is  appropriate. 
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APPENDIX  C-12 


DIAGNOSTIC  RECORD 
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1C  RECORD 


APPENDIX  C-13 


INSTRUCTIONS  FOR  THE  USE  OF  THE 
DIAGNOSTIC  RECORD  FORM 

The  purpose  of  this  form  is  to  help  you  document  your  analysis  so 
that  you  can  systematically  present  and  support  the  conclusions 
from  your  diagnosis.  The  form  should  be  filled  out  one 
deficiency  at  a  time. 

Step  1.  Record  in  the  "inquiry  data"  column  the  important 

data  that  you  gathered  when  asking  questions  about 
the  deficiency. 

Step  2.  Record  in  the  "problem(s)  identified"  column  your 

conclusion  as  to  what  the  problem (s)  was  based  on  the 
data  from  the  first  column. 

Step  3.  Record  what  the  deficiency  resulted.  This  is  a 

restatement  of  the  original  deficiency  you  diagnosed. 


Step  4.  In  the  cause  of  the  deficiency  column,  record  your 

conclusions  based  on  the  J~ta  from  the  first  column, 
as  to  what  caused  the  problem  identified  in  the 
second  column. 

Step  5.  Filling  in  recommendations  to  solve  the  problem  is 

optional.  If  by  experience  or  comfort  you  think  you 
have  a  good  solution  to  the  problem,  record  it. 
However,  the  purpose  of  the  diagnostics  is  to 
identify  problems  and  the  causes,  not  offer  remedy. 


055 


APPENDIX  D 


COMPUTING  SCORES  FROM  THE 
DEFICIENCY  CONSOLIDATION  WORKSHEET 


Computing  the  four  scores,  comprehensiveness,  criticality, 
effectiveness,  and  distribution  of  ratings  is  described  below. 

The  scores  are  computed  from  the  example  consolidation  worksheet 
and  Performance  Rating  Form) .  The  example  is  only  for  one 
section  of  the  worksheet,  enemy  courses  of  action,  and  one  rating 
period.  Also,  the  observer  participant's  weights  have  been 
recorded  on  the  worksheet  (identified  by  the  A)  from  the 
Information  Requirements  Profile.  ITEMS  ON  THE  WORKSHEET  AND 
FORM  ARE  IDENTIFIED  FOR  PROVIDING  THE  EXAMPLES.  IT  IS  NOT  PART 
OF  THE  COMPUTATION.  The  scores  based  on  all  the  worksheets  for 
each  rating  period  are  computed  in  the  same  manner  described. 

WHEN  COMPUTING  A  TOTAL  SCORE  AN  INFORMATION  ITEM  IS  ONLY  COUNTED 
AS  BEING  DEFICIENT  ONCE,  EVEN  IF  IT  WAS  DEFICIENT  IN  MORE  THAN 
ONE  DIMENSION. 

Comprehensiveness — total  score 

There  are  a  total  of  14  items  which  were  rated.  Four 
(identified  by  the  C)  show  no  deficiencies  in  any  of  the 
five  areas.  The  comprehensiveness  score  is  the  number  of 
non-deficient  items  divided  by  the  number  of  items  rated. 
The  result  is  multiplied  by  100  to  get  a  percent. 

Comprehensiveness  =  (4/14)  x  100 

=  .29  X  100 

=  29%  (this  is  probably  not  a  good 

score) 


Criticality — total  score 

There  are  10  items  that  were  rated  deficient.  They  had 
weights  of  40,  70,  80,  100,  70,  100,  80,  70,  40,  and  90. 
The  weights  came  out  of  the  column  identified  with  the  A 
for  items  identified  by  the  D.  The  sum  of  the  weights  of 
the  deficient  items  is  740. 

The  criticality  score  is  the  sum  of  the  weights  of  the 
deficient  items  divided  by  the  number  of  deficient  items. 

Criticality  =  740/10 

=  74  (Moderately  critical  items  were 
deficient) 
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Effectiveness — total  score 

There  are  4  items  not  deficient  (identified  by  the  C) . 
They  have  weights  of  70,  80,  80,  and  60.  The  sum  of 
these  weights  is  290.  The  effectiveness  score  is  the  sum 
of  the  weights  of  the  non-deficient  items  divided  by  the 
sum  of  the  weights  of  all  items.  The  result  is 
multiplied  by  100  to  get  a  percent. 

Effectiveness  =  (290/1030)  x  100 
=  .28  x  100 

=  28%  (not  a  good  score) 

The  three  scores  illustrated  can  also  be  computed  for  each 
dimension.  Computation  of  the  clarity  scores  is  described. 
Scores  for  the  other  deficiency  dimensions  are  computed  the  same 
way. 


Comprehensiveness — clarity 

There  were  only  10  items  that  were  rated  for  clarity. 
Since  four  information  items  were  not  received  by  the  G3 
plans,  they  could  not  be  rated  on  any  area  except 
timeliness.  There  were  seven  items  not  deficient  on 
clarity. 

Comprehensiveness  for  clarity  = 

(7/10)  x  100  =  70% 


Criticality — clarity 

The  three  deficient  items  for  clarity  had  weights  of  70, 
80,  and  100. 

Criticality-clarity  = 

250/3  =  83 


Effectiveness — clarity 

The  seven  non-deficient  items  for  clarity  had  scores  of 
70,  80,  80,  60,  80,  70,  and  90. 

Effectiveness-clarity  = 

(530/780)  X  100  =  68% 
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Distribution  of  performance  ratings 

The  distribution  of  performance  ratings,  while  not  a  score, 
represent  the  percent  of  time  a  specific  performance  scale 
value  was  selected  for  any  of  the  dimensions.  It  is  determined 
by  counting  how  many  times  the  item  was  given  a  performance 
scale  value,  divided  by  the  number  of  times  the  item  was  rated. 

From  the  example  Performance  Rating  Form,  enemy  courses 
of  action,  there  were  14  timeliness  ratings.  To  compute 
the  percent  for  any  performance  scale  value,  divide  the 
number  of  times  the  specific  value  occurred  by  the  number 
of  items  rated.  Multiply  that  result  by  100  to  get  the 
percent . 


Timeliness  Performance 


Scale  Value  Frequency 


Percent 


15  36 
2  0  0 
3  2  14 


4  3  21 

5  4  29 

The  line  between  performance  scale  value  3  and  4  represents 
the  benchmark  identified  by  the  observer-participant. 

The  other  areas  are  done  the  same  way,  except  they  only  have 
10  ratings.  Four  items  were  not  received  and  could  only  be 
rated  for  timeliness. 

The  scores  can  be  presented  in  graphs  or  tables  as  shown  on  the 
following  pages. 
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Performance  Rating  Form 
(part  2  of  2) 


in 


DEFICIENCY  CONSOLIDATION  WORKSHEET 


If  you  wanted  to  show  Ml  performance  on  a  rated  dimension  for  the  different 


This  shows  that  items  with  frequency  deficiencies  were  of  moderately  high  priority. 


If  you  are  interested  in  rating  distributions,  the  presentation  should  be  by  individual  dimension. 
How  well  did  Ml  perform  on  timeliness  during  the  first  rating  period? 


Timeliness 


This  figure  shows  that  on  90  %  of 
the  items  Ml  did  perfect.  The  other 
10%  of  the  items  weren’t  received 
(scale  value  of  5) 


If  you  were  interested  in  the  rating  trend  for  a  dimension  then: 


Completeness  1,  2,  3  are 


Scale  Value 


This  figure  shows  relative  flat  performance,  i.e.,  nothing  bad,  but  not  perfect  either.  The  trend  shows 
rating  period  3  to  have  the  most  variable  performance,  even  though  no  item  was  below  the  benchmark 
for  completeness. 
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What  can  be  shown  on  a  graph  can  be  shown  in  tabular  form  also. 


DIMENSION 

MEASURE 

1 

RATING  PERIOD 

2 

3 

8 

m 

Comprehensiveness 

80% 

85% 

90% 

w 

c 

1 

Criticality 

20% 

20% 

75% 

Effectiveness 

95% 

95% 

95% 

c 

« 

3 

Comprehensiveness 

90% 

100% 

100% 

U. 

Criticality 

40% 

0 

0 

Effectiveness 

95% 

100% 

100% 

RATING  PERIOD  1 
MEASURES 


DIMENSION 

COMPREHENSIVENESS 

CRITICALITY 

EFFECTIVENESS 

Timeliness 

60% 

50% 

75% 

Frequency 

80% 

30% 

70% 

Operational 

Perspective 

95% 

90% 

95% 

Clarity 

95% 

30% 

95% 

Completeness 

95% 

40% 

95% 
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